There is a story, perhaps apocryphal, of amateur archeologists making an astonishing discovery in the dunes outside Guadalupe, California. There, beneath the sand, lay what appeared to be the ruins of an ancient Egyptian civilisation: the remnants of sphinxes, the skeletons of chariots. It turned out to be the rubble from a Hollywood film set.

The film in question was 1923's The Ten Commandments, a silent-screen epic by Cecil B DeMille. No expense had been spared. The director hired 1,600 builders to construct a 10-acre site on the Guadalupe dunes, complete with a towering temple, 21 plaster sphinxes and a quartet of Ramses statues weighing 20 tonnes each. But when shooting wrapped, DeMille ordered that the whole place be bulldozed into a 300ft trench. There it remains to this day; a very Hollywood kind of ghost town.

Ghost towns are the eyesores in our midst, socio-economic roadkill; either imperfectly bulldozed or left to rot in the sun. The dictionary defines them as "deserted settlements, especially in the western US", which may explain their curious, symbiotic relationship with the American film industry. The birth of Hollywood, after all, coincided with the dying days of the wild west. It was a time in which Wyatt Earp worked as a studio script consultant and a collection of rusting former boom towns provided backdrops for anyone wanting to shoot a cowboy picture.

In the decades before it was thrown a lifeline by the tourist trade, the town of Lone Pine, California served as the base for over 250 westerns, including High Sierra and the peerless Bad Day at Black Rock, in which a one-armed Spencer Tracy confronts the locals of a misbegotten desert hamlet. Guadalupe, meanwhile, was a regular Hollywood destination until the 1940s, after which it became a hotbed of gambling, drugs and prostitution. And yet – like snowflakes, or grains of sand – no one ghost town is ever exactly like the other. To further muddy the waters, many confound the official dictionary definition by exhibiting stubborn, grubby signs of life.

Take the case of Bombay Beach, a haunting documentary about the lost souls who found a California ghost town and made it their home. History records that the film's backdrop enjoyed a brief heyday as a 1950s resort, playing host to Dean Martin, Jerry Lewis and Frank Sinatra. It is now parched, peeling and largely abandoned; a collection of derelict houses strung out under the telegraph wires. Even so, a smattering of misfits are still in residence. "The desert is very unforgiving," explains one. "You have to improvise in order to survive."

Bombay Beach – arranged in a series of startling, stylised tableaux by Israeli film-maker Alma Har'el – picks out three individuals in particular. It charts the travails of troubled, seven-year-old Benny Parrish, born to a family of flag-waving survivalists. It introduces us to teenaged Ceejay, on the run from gangland violence in Los Angeles and dreaming of a new life at college. And it shows us the ancient, incorrigible Red, a one-time oil man and unreconstructed racist who appears to subsist on a diet of alcohol and cigarettes alone. In this way the film paints a portrait of what the New York Times has described as the "three ages of man in post-apocalyptic America".

No doubt it's this human element that makes Bombay Beach so compelling. Without the presence of Benny, CeeJay and Red, Har'el's film might have risked becoming poverty tourism; an aesthetic pose, marvelling at the beauty of the junkyard or the sight of Main Street run to seed. As it is, there's something glorious about it.

"Well, yeah," allows Ha'rel. "But I actually have no real criticism for people who choose to go down that road: it's an intuitive, naive process of discovery. Ruins offer a lot of texture and composition. There's always a romance in the chaos. I mean, we don't get to see things that age any more. It's easier to buy something new than fix something old. So I can totally understand why people are drawn to the aesthetic of the ghost town."

Har'el would be forgiven for viewing America's ghost towns as exotic foreign ground. In fact, she claims that Bombay Beach felt weirdly familiar. "Israel, like America, started as a promise. So growing up, I felt like I'd been thrown into this place that was such a mix of history and obligation, hope, violence and tragedy. There's a settlement called Mitzpe Ramon, about three hours outside Tel Aviv, that's very like Bombay Beach. It's outside of society - inhabited by religious communities or people who are escaping their situation. So I don't think of these places as a purely American phenomenon. You can see that kind of post-apocalyptic urban wilderness all over the world."

Here in the UK, the role of the ghost town is largely filled by what writer Marion Shoard first christened the "edgelands" – those wild frontiers of our major urban centres, where the tyre yard sits cheek-by-jowl with the crumbling Victorian factory and the man-made lake without a name. "It's basically the logic of the American boom town, transplanted here," explains Michael Symmons Roberts, co-author of the book Edgelands: Journeys Into England's True Wilderness. "You can see the edgeland as a barometer for economic change. When the developers run out of money, what's left is a lot of half-built or half-empty buildings. They become the site for exciting and illicit behaviour. If you see these places on film or TV, it's always where the drug deal happens or the kidnapper is hiding out."

What drew Symmons Roberts to the edgelands – first as a child, then later as a poet – was the thrill of a place that is unmapped, unnamed, and often unwatched. To visit the edgeland is to step into a wilderness that's radically different from the "managed rural landscapes" that pass themselves off as wild and to catch a recent history in the raw. "It's a great perspective on our past, because it shows you the economic reality: the decay of what was once important and the green shoots of something starting up. It's like a litmus test of what we value and who we are."

But maybe there's a danger here as well. In shooting Bombay Beach, Alma Har'el inevitably leaves the town exposed. In naming the unnamed, Symmons Roberts risks destroying its mystery. "I heard about a reading group that organised a bus tour to visit their local edgelands," he confesses. "I wasn't sure what to think about that. I mean, we want people to appreciate these places without putting them in aspic. As soon as you slap on a preservation order, the whole thing's over."

Back in Guadalupe that process is already under way. There are plans (currently delayed by lack of funds) to excavate the set of The Ten Commandments and stick the props in an exhibition centre. And yet the thing about Bombay Beach – or the edgelands – is that they are not dead and buried under the dust. Their ruins grow over and their abandoned buildings are made use of. They exist off the radar and remain in flux. The obituaries, it seems, were premature.

"People say that Bombay Beach is about the death of the American dream," says Har'el. "But I think it actually shows the diversity and the variety of the American dream, and how that dream can twist and survive, even in the worst ghost town in California. So many people are in Bombay Beach because they are running away from something. But they are living their lives and those lives are inspiring." Post-apocalyptic, perhaps, is just another term for rebirth and renewal.

If Bombay Beach has a patron saint, it can surely be found in the stooped, weathered form of Red, a roustabout son of the Great Depression, still living in hard times. At one stage, Har'el shows Red on what can only be his deathbed, hooked up to an oxygen mask and breathing his last. But by the end of the film he is somehow back on his feet, all set to fight (and drink, and smoke) another day. "I never say goodbye," explains Red in a line that could serve as the motto for any self-respecting ghost town. "I say so long, so long, hope to see you again."

• Bombay Beach is released on Friday 3 February