A wooden cabin by a lake in the middle of rural Ireland. A freezing midwinter morning. Four musicians are in the living room. A bass guitar wobbles between two alternating notes. A complex drum fill crashes for half a bar, then stops. Two atonal synthesiser stabs, each from a different keyboard. From somewhere else in the cabin, a man's voice moans: "Again." The musicians go through the cycle again: bass wobble, drum fill, synth, synth. "Again," the voice wails, from a  different room. Bass, drum, synth, synth.

I'm watching the band play on a monitor from the room next door, huddled with other technicians out of shot. The man shuffles past us and shouts through the wall: "Again!" Bass, drum, synth, synth. He is wearing a giant fibreglass head with big blue eyes and a severe parting painted on to it. The technicians are trying to keep their faces straight. The man opens the door and pokes his head into shot: "Again!" Underneath the head is Michael Fassbender, though at this stage, it can't be proved.

We are on set of the movie Frank, an eccentric comedy about an eccentric rock band led by a man who has wandered off the spectrum of eccentricity and into mental illness. In a movie market crowded with prestige biopics, fictional comedies, concert movies, documentaries and mockumentaries, Frank is aiming for something a bit alternative. A lot more alternative, actually, as this morning's scene suggests. The band in the movie goes by the "so edgy no one can even pronounce it" name of the Soronprfbs. They have come to this muddy corner of County Wicklow to record their new album – spurred on by their obsessive, inscrutable, fibreglass-headed leader. They have been here a week in real life and 18 months in the movie, which explains why Domhnall Gleeson, the keyboard player, is sporting a long straggly beard.

Indie aficionados of a certain vintage will recognise the head as a homage to Frank Sidebottom, alter-ego of Mancunian musician Chris Sievey, who died in 2010. Sidebottom was a genuine British eccentric, part-punk provocateur, part-end-of-the-pier parody. His gigs were unpredictable, shambolic affairs, often involving audience participation, puppets, references to Sievey's home village of Timperley, and the mauling of popular hits such as I Should Be So Lucky and Anarchy in the UK. He sang along to a lo-fi synthesiser in a cheery nasal whine, was fond of using the word "bobbins", and tended to end every song with the phrase: "You know it is, it really is." It is hard to imagine an act less geared towards the big time, which is exactly what his fans liked about him.

Understandably perhaps, certain adjustments were necessary for Frank, the movie. It was co-written by Jon Ronson, inspired by his time in Sievey's band in the late 80s, but this Frank is American and present-day, explains director Lenny Abrahamson from beneath a woolly cap and several layers of outerwear.

"We've moved so far from Jon's original memoir that began the process," he says, over a rapidly cooling lunch on the catering bus. "What you've got left is a kind of debt to Frank Sidebottom in the aspects of the sensibility, and the head. I hope it's a tribute to him, but it's also inspired by lots of other outsider musicians and strange, creative but ultimately damaged individuals. So we looked at Captain Beefheart and Frank Zappa and Daniel Johnston, lots of people like that." Beefheart notoriously held his band under virtual house arrest in California in the 60s to make his magnum opus Trout Mask Replica – but at least they were somewhere warm.

There is a balance to strike here, or is it a contradiction? Like a sprawling desert, the "mainstream" inevitably subsumes all things "alternative" over time, in both independent music and cinema. Cult musicians such as Sidebottom, Beefheart and Johnston are guarded by their fans like endangered species in a nature reserve. By opening up these backwaters, there is a danger Frank could earn the enmity of the very pop connoisseurs it hopes to attract. In fact, Frank's story addresses exactly this dilemma: between artistic integrity (and obscurity) and commercial success. The band are dragged toward the latter by Gleeson's ambitious new keyboard player, Jon. This Twitter-savvy newcomer secures the band a gig at SXSW, the Texan music festival where the underground traditionally goes over. It turns out to be their undoing.

"Everybody else in the band is madder than Jon," explains Gleeson between takes, in the character's middle-class English accent (if he slips into his native Irish between takes, his vowels start going, he explains), "but underneath the politeness, he has ambition. He puts pressure on the band to be more mainstream." He's smoking a cigarette through a holder so as not to ruin his straggly beard.

You couldn't accuse Frank of "selling out", exactly. It's not a big budget film, and there is an aura of playful experimentation that seems to be sustaining the cast through the mud-spattered adversity. Abrahamson, an emerging force in Irish cinema, has already proved his outsider credentials. His debut, Adam & Paul, followed two comically hapless low-lifes around the more squalid areas of Dublin, while 2007's Garage centred on a simple-minded petrol station worker. Frank is less of a departure for him than it looks, he says, especially in terms of the tone, which he describes as "a celebration of a certain kind of delicate slapstick, which I think has massive potential to express emotion. Because the nature of this band is to muddle through chaotically, it means we don't have to be po-faced about it. We're not making Control here."

There is also something heartening about Frank's spectacularly un-commercial decision to cast a face as recognisable as Fassbender's, then conceal it beneath a fake head. In the movie, Frank even wears the head in the shower.

"It's pretty comfortable now. I'm used to it," explains Fassbender during a break. He has taken the head off and it really is him beneath it. "It was kind of weird in the beginning, the way it sits on your head. You feel a little bit vulnerable because you've not much peripheral vision. You can only see straight in front of you. You can't hear that well either. But it's quite liberating as well, in the way that masks are. It's amazing how many emotions that face manages to convey. It can look vulnerable, it can look sinister, at times it looks confused."

Before this, Fassbender was shooting in Texas with Terrence Malick. A few months hence, he will receive an Oscar nomination for 12 Years A Slave. Why is he doing Frank? "The script just made me laugh out loud," he replies, admitting he had never heard of Frank Sidebottom. "I thought there were also poignant moments there. It's a very unusual story, a real standalone. If it's refreshing to read, it's hopefully refreshing to watch."

There's another way in which Frank boosts its alternative credentials, which really is refreshing: the band is a real band. The actors actually play the music you hear in the film. That explains why the Soronprfbs' drummer is Carla Azar, a sometime member of Jack White's band (you can tell just by that half a fill that she's a professional). The bass player is François Civil, a French actor/musician. Completing the lineup is Maggie Gyllenhaal on vocals, Moog and theremin. The latter is not an easy instrument to learn, she says. "It's basically like playing electricity." Gyllenhaal plays Frank's protective girlfriend, Carla. Acting opposite an expressionless head is no big deal. "It's not entirely foreign to me. I've had experiences like that in my life."

Like many actors, Fassbender, who sings and plays guitar and keyboards in the film, admits he once harboured fantasies of rock stardom. "I wouldn't call it a band," he says, recalling his efforts as a 17-year-old. "It was just me and another guy on guitar. We could never find a drummer or a bass player. We never even got as far as having a name. We were both called Michael so we could have been Michael and Michael." No wonder he turned to acting.

Before coming here to pretend to rehearse, they spent three weeks actually rehearsing in Dublin, under the guidance of Abrahamson and his composer Stephen Rennicks. It wasn't quite a Beefheart-style drilling, but they threw themselves into it, says Gleeson. "Every time Stephen and Lenny stepped out of the room, Michael would pick a song that we all had to try and play while he sang over it." The cast are enjoying being a band so much, they're talking about going on tour when the film's finished.

So what do the Soronprfbs sound like? Abrahamson's vision was "pathologically eclectic". Suggestions from the cast include Captain Beefheart, Can, Sigur Rós and the Beta Band. "Somebody said one song sounded like Jim Morrison," says Fassbender, "but then the next one sounded like the Sex Pistols. We're never supposed to be good pop music, we're supposed to be odd music that strikes a beautiful chord in some kind of way, but remaining sort of … not shit music exactly, but let's say … hard to market."

Is there a difference between genuinely uncommercial and marketably uncommercial? Is it the difference between Frank Sidebottom and Frank the movie? Even a mid-budget movie costs a lot of money to make, and needs to make it back whereas financial success was practically an obstacle to artistic achievement for most of the outsider musicians Frank celebrates. Sievey died practically penniless. He only received a proper funeral thanks to a last-minute fundraising appeal by fans. Who knows what he would have made of Frank the movie. Doubtless he would be flattered, but he would be even happier to know that a life-size statue of Frank Sidebottom now stands in Timperley.

All the bobbins: The beginner's guide to Frank Sidebottom

Frank Sidebottom, resplendent in a fibreglass head, shuffled onto the 80s pop scene like a postmodern riff on George Formby or Norman Wisdom, full of guileless chatter and novelty tunes. He hailed from Timperley on the outskirts of Manchester, claimed to still live with his mum and sang songs with titles like Christmas is Really Brilliant and I'm the Urban Spaceman. Frank's bandmates included Mark Radcliffe and Jon Ronson. Chris Evans earned a crust as his driver, while Caroline Aherne road-tested her Mrs Merton alter-ego on the entertainer's Radio Timperley show. Everyone associated with Frank, it seemed, was destined to grow and to prosper and to leave him behind.

Intriguingly, Frank Sidebottom was never seen in the same room at the same time as the musician and comic Chris Sievey. And when Sievey died in May 2010 at the age of 54, Frank is believed to have died along with him. Xan Brooks

• Frank premieres at the Sundance London film and music festival on Friday 25 April. It will be on general release in the UK from 9 May.