The 70-year-old comedian recently starred opposite the drama's Dame Maggie Smith in Quartet.
While he said he was unsure how he could fit in the show, he added that he would love to reunite with Smith, who plays Violet Crawley in the drama.
"I loved working with Dame Maggie Smith. She's hilarious, so funny in between takes and then they say action and she can turn it on like that," Connolly told Heat at last night's BAFTAs ceremony (February 10).
"She's amazing and the things she comes out with, real acidic comments, it's brilliant. She is a bit like the character on Downton."
He added: "I'd love to work with her again, I'm not sure how I'd fit into the Downton world, it's maybe not quite right but it would be a laugh I guess."