The fastest-selling cultural product in history was created by people you've probably never heard of. While this year's Oscars honoured films in which the movie business sweetly congratulates itself on its own birth – The Artist, Hugo – the most dollar-hoovering entertainment release ever is not a film, still less an album; it's a video game. Coming out last autumn, Modern Warfare 3 – a blockbuster military shooter made by a Californian game studio called Infinity Ward – took just 16 days to gross $1bn, beating by one day the previous record set by a film about blue people in space. And it wasn't a freak accident. Global annual sales of video games now dwarf cinema box-office and recorded music: in 2010, games grossed $56bn, film tickets $32bn and music $23bn. (The film industry as a whole still made more, at $87bn.) Even social games on Facebook are enormous business: Zynga, the firm behind Farmville and Words With Friends, is responsible for 12% of Facebook's revenue. Hollywood is old-school now. And one company in particular has played a pivotal role in this media revolution over the past decade: Rockstar Games.

Rockstar's banner Grand Theft Auto series has sold a total of 117m copies. And it's a cute irony of cultural globalisation that the most convincing digital simulation of New York yet made was built by a gang of Scots. In 2008, the $1bn-grossing video game Grand Theft Auto IV recreated in spectacular fidelity Manhattan and its environs as the setting for the adventures of Niko Bellic, an eastern European migrant intent on upward social mobility in the criminal underworld. Later this year, Grand Theft Auto V – whose recently released teaser trailer has, like that for a hotly anticipated film, already attracted millions of views and countless pages of badly spelled fan speculation on the internet – will move the action to a virtual Los Angeles. Yet all the main episodes in this monster fun franchise are created in the UK by Rockstar North, an Edinburgh-based studio that began as a plucky startup in the bedroom-coding home-computer revolution of the 1980s.

Once upon a time, Rockstar North was DMA Design, founded in 1988 by a group of friends in Dundee. Their first big hit was Lemmings, a puzzle game in which you guide a troupe of the suicidally trusting furry creatures through a series of sadistically booby-trapped levels. Lemmings became a guilty hit in offices around the country, and with sequels and spin-offs had by the early 1990s sold more than 20m. But the cartoonish violence of squished lemmings transmuted into something much edgier when DMA brought out their next game, Grand Theft Auto.

The first GTA began production in 1995 under the working title Race'n'Chase. At first you could play either a policeman or a criminal, but the team soon realised that enforcing the law was not as much fun as breaking it. Dave Jones, one of DMA's founders, now explains: "It was just so much more fun doing all the crazy wrong stuff. There was no way we could get as much fun from being the good guys. Eventually, we just dropped the two-sided approach and fully embraced the dark side." Visually, GTA was essentially a cartoon, with a vantage point like that of the satellite view of streets you get today in Google Maps. You drove your car around the city grid, stopping off at payphones to receive instructions – rob this bank, destroy that vehicle – and mowing down cops or civilian pedestrians, whose tiny pixilated forms would squelch bloodily under your tyres. Bonus points were awarded for killing an entire orange-robed conga line of Hare Krishnas.

Max Clifford was hired to advise on the PR for the game's 1997 release. Cannily, he advised DMA to feed the tabloids the most outrageous details possible. "It was scary and impressive how he laid out his plan to manipulate the media and the politicians," Jones says. "It culminated in a two-hour feature on breakfast TV debating the game. At this point, the politicians lambasting the game had not even seen it – I think they were disappointed when they did, given the cartoony look." The tabloids duly issued calls to ban this sick filth, the British Police Federation said it was "sick, deluded and beneath contempt", and the game became a hit. Grand Theft Auto the countercultural phenomenon was born.

At the time, DMA's games were published by BMG Interactive, a London division of the German music group, where a young Brit called Sam Houser took a close interest. When I talked to him on the release of Grand Theft Auto II in 1999, Houser was bullish about the controversy over the first game. He recalled talking to the New York Police Department, who apparently didn't mind that youngsters were killing cops in GTA: "Well, you know what?" he recalled them saying. "There's a lot of people out there trying to kill cops, and we'd rather they did it in your game than on the street."

Houser is now president of the New York-based game-publishing powerhouse Rockstar Games, which he co-founded in 1998 with his brother Dan, among others. DMA Design became its wholly owned subsidiary Rockstar North, one of several Rockstar studios around the world. (Max Payne 3, due out in May, is created by Rockstar Vancouver.) And Houser himself has been the GTA games' executive producer since the third outing, 2001's Grand Theft Auto III, which was a technical revolution for the series – it no longer offered a flat, aerial vista of the city, but a perspectival, street-level point of view. The carjacking, shooting and running-over was close-up and visceral, and yet the game was also very funny, and had a rare depth beneath the lurid mischief. "It is a beautifully designed game with glorious cascading systems," says Ste Curran, creative director at British game studio Echo Peak and host of the video games radio show One Life Left. "The location is fun – it looks like a city, feels like a playground – and events unfold differently on every replay. That's what keeps the game interesting."

And then, of course, there are the prostitutes. Invite one of GTA III's streetwalkers into your stolen car, and in exchange for some cash you would regain some of your "health", after a chaste interlude of bouncing suspension. One designer, however, who tried beating the prostitute to death immediately after the transaction, got his money back and kept the health increase: win-win. GTA III was released on another wave of controversy about how it was the game in which you killed prostitutes. Some commentators pointed out mildly that you didn't have to kill prostitutes; but the designers did make it possible – as they did not, for example, make it possible to rescue kittens stuck up trees. But then, that wouldn't have fit with the game's swaggering aesthetic, what Curran describes as its "perfectly pitched Molotov cocktail of pop culture and grimy, streetwise theft and thuggery". An employee at the time, who spoke on condition of anonymity, remembers that while the atmosphere within Rockstar was playful ("table-tennis tables and beanbags and toys on the desk and all that kind of Generation Y stuff"), the top people were also "very driven". Their laudable "passion for the games" also led to office shouting matches and a "face-time culture", with people routinely expected to "stay late even if they'd finished their work".

The next game in the series, Grand Theft Auto: Vice City (2002), was a milestone in Houser's ambition to make "cinematic" video games. Stylistically, it was a loving tribute to the film Scarface and the 1980s TV series Miami Vice. Ray Liotta, Peter Fonda and Dennis Hopper were cast as voice-actors. Vice City was the first successful "period" video game, showing that the medium had become mature enough to riff on a concrete historical period rather than a pseudo-Tolkein alt-middle ages or a quasi-Blade Runner sci-fi future. Dave Jones, one of GTA's original creators, had left DMA after the release of the second game ("It was not my company any more"), and is now a creative director working with several studios. So what does he think of how Rockstar brought up his baby? "I thought it was great," he enthuses. "GTA III, and especially Vice City, for me were the pinnacles of the series. They kept a lot of the non-serious side of the game, which I personally still feel is the best treatment for it."

The sequel that followed, Grand Theft Auto: San Andreas (2004), was less whimsically amusing, but even more provocative, set as it was among drug and prostitution gangs in LA during the 1990s. It was progressive in having a black lead character, almost unheard of in video games at the time. But San Andreas also became the most notorious game yet of the series. During the game's development, as a new book on the history of the GTA games, David Kushner's Jacked, relates, Houser had been pushing hard to feature porny scenes, including representations of "blowjobs", "dildo sex" and "whipping". He was eventually convinced by his partners that this would be commercial suicide, since big retailers such as Wal-Mart would refuse to stock it. Right before the game's launch, the offending parts from the game's complex code were hastily "wrapped": they would never appear when you played the game, but they were still there, deep in the data on the discs. Inevitably, an enterprising hacker discovered them and released on to the internet a software patch to let other players unlock them, too. This became known as the "hot coffee" modification – so called because in the deleted scenes the main character's girlfriend invites him into her house for "coffee". Moralists on both sides of the Atlantic went apoplectic.

Rockstar's anonymous employee vividly recalls the epic all-nighter when Rockstar was scrambling to re-release a new version of San Andreas purged of the sex scenes: "Pretty much the entire company got pulled in to help with the testing process in order to get a new version of the game out as quickly as possible to appease the American censors. That meant we had to work round the clock in shifts, sleeping on sofas to sustain us through the night so we could shoot imaginary gangsters and hoodlums in the head."

When Bobby Kotick, the amiable CEO of rival major publishers Activision Blizzard, was asked at a 2008 industry conference whether games had to be as violent as Grand Theft Auto, he defended Rockstar. "Fifty per cent of the audience that plays games is over the age of 18," he pointed out. "We're a broad-based medium today, we're going to appeal to the broadest possible consumer base, and you're going to see all sorts of product, and that's going to include gratuitously violent product." In any case, violent games have only ever been one part of the wider video-game story. As well as the Call Of Duty series of military shooters (which includes the Modern Warfare sub-brand), Activision Blizzard also publishes the hugely successful elf-bothering online roleplayer World Of Warcraft. Just as with films, different video games are made for different audiences. Another big publisher, EA, meanwhile, produces popular sports games featuring snowboarding or football and the life-simulation game The Sims.

Rockstar, in particular, wants its games to be taken as seriously as films are. It certainly pours a comparable level of resources into production, pushing games into what Ste Curran terms a "mega-budget era". For GTA IV, say, the budget was around $100m, it took three years to create, and used a cast of 861 actors speaking 80,000 lines of dialogue. Large video-game productions now have staff (like permanent film crews) numbering in the hundreds; they commission symphonic musical scores, and continue to poach Hollywood talent. (The actor Mark Hamill, once a fresh-faced Luke Skywalker, has been playing a blinder as the Joker in the recent Batman games.) A new mini-wave of noir-ish detective games, such as the French designer David Cage's Heavy Rain or Rockstar's own LA Noire, is pushing the technique of "motion capture" to new heights, recording actors' facial expressions as they speak and then applying those movements to digitally created physiognomies. (This is how Gollum in the Lord Of The Rings films was created.) For the moment this has the effect of driving video games even further into what some digital aestheticians call "Uncanny Valley", a strange no-man's-land where the more realistic an artificial person looks, the eerier its niggling departures from reality feel. But David Cage, for one, has predicted that fully "photo-realistic" video-game characters will be possible in around six years' time.

So what kinds of pseudo-films are today's most successful video games? If the immensely slick Call Of Duty series were a film director, it would be Michael Bay, all deafening destruction and comedy geopolitics. (The last two Modern Warfare games were predicated on the notion of the Russians mounting a land-based invasion of the US and Europe.) If Rockstar's Grand Theft Auto series were a film director, it would be Martin Scorsese – or at least that's who it really wants to be – but it takes more than an obsessive fan's line-by-line recall of GoodFellas to make art. Reviewing GTA IV for the Wall Street Journal in 2008, the novelist Junot Díaz, a long-time fan, ridiculed the media hype that had compared it favourably to Coppola's The Godfather. "Like the pulps that are part of its narrative DNA," he wrote, "GTA IV operates in broad strokes, crude characterization and over-the-top stereotypes."

"Video games tell stories badly," says the game designer and critic Ian Bogost, a professor at the Georgia Institute of Technology. (He designed the Facebook game Cow Clicker, a satire on Farmville-style games that itself became a surprise hit.) Even so, Bogost says, bad game stories are "charming", and perhaps even necessarily bad: "Maybe video games are meant to help us shed our obsession with storytelling, show us all the things between the story, like wandering around virtual space and exploring it like virtual tourists."

Virtual tourism is, indeed, the aspect of the Grand Theft Auto games that has been much more influential for the medium as a whole than their gangster-movie envy. They made popular and compelling an "open-world" style. If you want to progress through the game's scripted narrative, you must accept specific missions of telegenic spatter-mayhem; but that is not all you can do. Instead, you may just wander around and soak up the sights, which these days are impressive. That video games now provide a place where you can go to relax is itself a sign of their rapidly burgeoning capacity for rich simulation. "We have all been born 100 years too early," Dave Jones laments wryly. "I would love to build and play in the kind of environments we see in movies like The Matrix and Inception." But, he notes, we are still in "the stone age of gaming technology". Jones's own most recent project, APBReloaded, is like a networked GTA: the other people in the city are not scripted, artificial characters, but human beings playing over the internet. His vision for the future is like this, only more so: "I want to be playing in an even more realistic GTA-style environment," Jones says, "with 1,000 other real players in the city." And so the impish simulation of antisocial behaviour promises to become ever more sociable.

You may not have experienced a Grand Theft Auto game yourself, but you can hardly pass the day in a modern city without seeing someone playing a video game on their laptop or smartphone. And Rockstar have had a crucial role in gaining mass cultural acceptance for the medium ever since they inspired other game-makers, according to Ste Curran, to emulate GTA's "potent, lucrative blend of mainstream cool and commercial success". Now video games have indeed become as mainstream as music and the movies. We live in an age of ambient play. And perhaps it is not just a coincidence that the recent video-game trend of repurposing cities as zones of anarchic fun has coincided with developments in the wealthy real world such as urban riots and the Occupy movement. If so, roll on Grand Theft Auto V: we can still reclaim the virtual streets, if not the real ones.

• Steven Poole is the author of Trigger Happy: The Inner Life of Videogames (Fourth Estate, £7.99).