So, Downton Abbey. You're well out of it, right? "Well out of it? What do you mean?" Dan Stevens knows exactly what I mean, but he is too loyal to say. It's gone off the boil a bit, I say, since you left. Did he get out of the UK's most popular costume drama in recent history at just the right time? "I felt it was the right moment to leave and I went with that feeling," he says, earnestly. "It is what it is. It was a fun thing to be a part of. I like the way people enjoy the show with a mixture of affection and humour."

This is an excellent description of the peculiarly British phenomenon that made Stevens' name as Matthew Crawley, the heir to the Downton estate. His untimely demise in a car crash had viewers choking on their mince pies at the end of the Christmas special two years ago. Despite memorable turns in BBC adaptations of Sense And Sensibility (where he played Edward Ferrars, obviously) and The Line Of Beauty (where he played the lead role of Nick Guest), until now this is easily the role for which the once floppy-haired and Hugh Grant-esque Stevens has been best known.

The death of cousin Matthew led to a barrage of complaints on social media: "Downton Abbey, you have ruined my Christmas Day"; "I don't even want to discuss what happened"; "I am 100% done with this show. I cannot breathe." Stevens, 31, was later forced to apologise and said, "What emerged is that it's an unwritten rule that you're not supposed to die on British television on Christmas Day."

Now with his Broadway debut behind him (he starred opposite Jessica Chastain in The Heiress, based on a Henry James novella and was praised for his "beguiling charm" by the Hollywood Reporter, but "shiny, well-spoken and lacking in discernible undercurrents" according to the New York Times), Stevens has four feature films out this autumn, including The Cobbler, a comedy with Adam Sandler, Steve Buscemi and Dustin Hoffman; and a turn as Sir Lancelot opposite Ben Stiller in Night At The Museum 3. Downton is a distant memory. "Have I had a good time [since leaving]? Yes. Would I have enjoyed it if I had stayed? Possibly. Possibly not. Without getting into contract stuff [he left during contract-renewal discussions], it's no secret that's what it was. It was a big choice to make at the time. But quite an exciting one."

Downton has 12 million UK viewers and has sold to more than 100 countries. American fans hold Downton viewing parties, where they dress up as characters from the show. When did he realise that Downton was massive? "At the time it happened quite gradually, and then suddenly very quickly. It became weirdly popular in Spain. Like the most watched foreign thing on Spanish TV in fifteen years or something like that." More recently, he discovered that his co-star Ben Stiller was a huge fan. "That really made me smile. He wanted to ask a lot of questions, the way Downton fans do a lot of the time." On set, the two of them used to play a multiple choice quiz called QuizUp. "It's a bit like Trivial Pursuit. He [Stiller] is really into that and he is quite competitive. We challenged each other at Downton Abbey. He would always beat me."

Two years on and his decision to leave is paying off. Having seen The Guest, a clever suspense thriller with a playful sense of humour, and A Walk Among The Tombstones, the new Liam Neeson blockbuster, I would wager that Stevens is styling himself as a new De Niro; failing that, he's auditioning to be the next James Bond or Jason Bourne. Either way, it would be hard to choose parts further away from lords, ladies and dowagers. In A Walk Among The Tombstones, he plays a Nabokov-reading, fine art-collecting New York drug dealer who enlists the help of Neeson's retired cop to solve the murder of his wife. In The Guest, he plays a preposterously buff and aggressive special operations vet returning from Iraq, visiting the grieving family of his dead team-mate.

We meet at a photographic studio in east London. His accent is exactly like Matthew Crawley's: soft and well-spoken, Cambridge-educated without being annoyingly posh. He is polite, laughs easily but has a hint of shyness. He teases me about my coverage of Downton Abbey. (I blog the series for the Guardian, and not always in the most complimentary terms – although, Stevens agrees, most of my sneering could be termed affectionate. Slightly less so as time has gone on.)

Stevens has clearly lost some weight, and the one-time baby face has transformed into something more piercing and sharp; he looks as lean and mean as his characters. Off-camera, he even looks slightly gaunt. He seems, in fact, like someone who has been living in New York (and living the New York lifestyle) for some time and who no longer consumes dairy products. Which is exactly what he is. ("I'm not vegan. I'm trying to eat healthier. And I've never really liked dairy," he says, apologetically.)

It's hard not to suspect that the absurdist developments with the Matthew Crawley character played a huge part in his decision to move to the US. Towards the end of the second series, Stevens was required to play a man who had been paralysed and rendered impotent by his activities in the first world war. Then, suddenly, as if the script writer had just remembered it would be difficult to have a leading man who could neither walk nor impregnate anyone, there was an episode where he was forced to rise from his bath chair and exclaim, "I felt a tingle!" That cannot have been easy to act. "It certainly went all over the place," he laughs, "It had some crazy twists and turns and that was fun."

All the same, Stevens was walking away from a hit. Wasn't that a brave – even reckless – move? "It was the first time I had done a long-running TV thing. Some people are better suited to that than others. I felt like a change. We had just had our second child, who was three months old when we moved to the States." Stevens is married to singer Susie Harriet and they have two children, Willow, four, and Aubrey, two. "I didn't take a decision like that lightly, taking my family overseas. But those decisions can yield interesting results."

Although he has to hold (flawless) American accents in both films, The Guest, in particular, is a bold choice – an entertaining, self-referential psychological thriller. The director is Adam Wingard, whose last film, You're Next, a slasher movie, became an underground hit, costing $1m to make and taking $25m at the box office. In another era, it's a part that might have been played by Schwarzenegger or Stallone. Stevens needs to look ripped enough to be a plausible special operations soldier – and to carry a scene where he steps out of the bathroom wearing only a very small towel, six-pack rippling.

Obviously Stevens is not going to Google "Dan Stevens naked torso", but others have (7.7 million results on Google), racking up hundreds of thousands of hits for The Guest's trailer. What does he make of this phenomenon? "For a start, I was not aware that it was such a phenomenon. As I did not have those statistics to hand." He coughs and seems embarrassed. "That's amazing. It's a great trailer. The teaser was amazing and then they expanded it... I'm glad people are intrigued by it." How does it feel seeing himself on screen looking like that? Again, embarrassment. "It's a key part of the humour of the film. And it was important to get that across." He means mean he finds it funny to see him,self like that? "Er, yeah. Especially in the context of the film. When you see how that image plays into the narrative, it is funny." Fair enough. Say more and you spoil the film.

"I met some special ops soldiers and I was very gratified to find they aren't all huge guys. They're lean, athletic. They're not all like Superman. A lot of them looked like me. Because if you are big and bulky, you can't handle your pack or your machine gun. Psychologically, they were very interesting to hang out with – they don't talk a huge amount about what they've been up to.

"It was shot in New Mexico and I went out early for a month of military training," he says. "I had to learn how to handle a Glock [pistol] and how to take guns apart. I hadn't done that kind of thing before. It was so far outside my ken. Really exciting." Did they use a body double for the shower scene? "No," he laughs. "I worked very hard. I'm not a natural gym bunny and there was a lot of pain involved. I was training for four hours a day for a month, and then we shot for two months and I continued that training. Aesthetically, it was surprising. I have never looked like that in my life. My wife was very happy."

Now that he lives mostly in New York – his daughter has just started at school – is he the sort of person who works out with a trainer every day? "I'm not suddenly evangelical about it. If I had to do it for another role, I could do it again. I feel like I'm ready for anything. I had been coasting in a state of 'not much gym' for five years. And suddenly 'lots of gym'. And a bit of martial arts mixed in. And discipline. I'm not always good at that."

Discipline is something he claims to have struggled with in the past. He is coy about his upbringing and education. He was adopted at seven days old, and raised in Essex and Wales. His parents, retired teachers, live in Sussex. He has a younger brother, adopted from another family. "I've been a very lucky product of the process of adoption and ended up with really wonderful parents. It's an extraordinary thing, adoption." Has he sought out his birth parents? "Not really actively. I've done as much research as I need to do to satisfy my own curiosity." He speaks fondly of his parents and his childhood (his mother taught science and was later a primary school teacher; his father taught business studies and economics), but there's a suggestion of some difficulties. At the age of 10, he was sent to boarding school at Tonbridge in Kent.

"They are strange things, boarding schools. A peculiarly English thing. Putting 60 boys in one house for 12 weeks is definitely an odd thing to do. I think I probably did benefit from the discipline. I always enjoyed having something to kick against. And the institutionalised thing... I have never been very good at it. But I think in life it's good to have a sense of what that means." Are you saying you were hard to handle as a child? "I don't know. But, yeah, probably. I was never very happy in institutionalised environments, and you have to get better at it if you're in boarding school. I was one of those boys who smoked at school because it was illegal. In university, as soon as it was allowed, I gave up smoking."

But he can't have been all that rebellious as he got into Cambridge to study English Literature. He credits his teachers for that – and for pushing him into acting. "There are quite a lot of teachers who say, 'Shut up or get out', and any kind of sensibility doesn't get nurtured. It's a really good thing for a child who is very distracted in a classroom to take them and say, 'Do that. But don't do that.' Looking back, I can see the people who spotted that in me and I am incredibly grateful to them. I didn't really realise that was happening at the time."

At Cambridge he was good friends with comedians Mark Watson, Tim Key and Stefan Golaszewski. "I was doing Footlights at the same time as them. We were all Chris Morris fans – that sort of dark comedy rather than the 'let's put on straw boaters' kind of comedy. Then I did a couple of Shakespeare plays. It was funny in the end that the two serious plays I did led to me getting my first jobs. So I fell into this classical training." Sir Peter Hall cast him opposite his daughter Rebecca in a touring production of As You Like It. The production went on to tour the US and won Stevens a coveted commendation in the 2004 Ian Charleson Awards. "It was the first time somebody had paid me to do what I really love doing, and it felt momentous for me. I still feel that at times on certain jobs: 'Wow, I'm really doing this – this is really cool.' I'm doing what I always dreamed of doing."

He is one of life's enthusiasts, which is what led him to say yes when asked to judge the Booker prize in 2012, the year Hilary Mantel's Bring Up The Bodies (the sequel to Wolf Hall) won. It was perhaps surprising casting, and he was widely presumed to be the inspiration for Tobias, a character in Edward St Aubyn's latest novel, Lost For Words, a farce based on the judging of a fictional book prize. I get the sense the judging might also have played a role in Stevens getting fed up of his life in England. "It was a crazy thing to say yes to. But I'm really glad I did. It has taken me a while to come round to thinking that. The points at which you are most regretful come exactly midway between the judges' meetings. You have these check-in meetings. And then you have the mountain that faces you when you get home and you're surrounded by 150 books and you just want to cry. You have six or seven months to read them. And the long-listed and short-listed ones you read multiple times. I read Bring Up The Bodies four and a half times."

He won't be drawn on my theory that he's on some mega-ambitious quest to take over Hollywood. If he is, then phase one is a success. "It's more about challenging myself," he says. "Seeing how far I can push the believability in my own mind. Then your confidence builds from that, in terms of what you attack and the challenges you set yourself."

He still has time, however, for some of his old Downton mates, and meets up with the surviving cast members on a regular basis (he won't say who). "Oh, yes. Brunch with Dame Maggie every Thursday at the Wolseley," he jokes. So what's the next absurdist plot twist, then? "No, seriously. They are good at keeping the story locked down. They won't tell me anything." Like I say: well out of it.

• The Guest is released on 5 September.