But not everyone can be a Lord Grantham or a Carson. For many - whether they chose to leave or were written out - their fortunes changed after exiting the ITV drama.
So after the news that Dan Stevens has joined Beauty and the Beast, Digital Spy looks at the careers (so far!) of 9 actors who have bowed out of Highclere Castle...
1. Ed Speleers
After starring in series three, four and five of Downton, Ed Speleers left Jimmy Kent behind for another role in a big budget British TV drama.
We most recently saw him in the part of Edward Seymour in Wolf Hall, and he will also star in upcoming films Remainder and Howl. Most prominently, he's been cast alongside Johnny Depp, Anne Hathaway and Helena Bonham Carter in Alice in Wonderland: Through the Looking Glass.
2. Amy Nuttall