Sweet Billy Pilgrim are an actual English genre-spanning band that adopted their name from the main character in the Kurt Vonnegut's novel Slaughterhouse-Five.
The exterior of the theater where Sweet Billy Pilgrim plays is the same theater in The Work Outing (2007) where 'Gay, the Gay Musical' was performed.
Roy wears a T-shirt that says 'LET'S SAY YOU'VE GONE BACK IN TIME'. Chris O'Dowd was in Frequently Asked Questions About Time Travel (2009), a film where he and his friends travel in time.
In Douglas' video, after he says "Every star you can see in the night sky is a wish that has come true", five bell notes form the alien motif from Close Encounters of the Third Kind (1977).
Roy's work t-shirt in this episode is a ginger cat, wearing a blue top, playing an electric piano.