When David Merlin (David Niven) and Polly (Ginger Rogers) leave the New Years Eve party, the amount of confetti on his jacket changes dramatically from when he talks to the lady by the door and when he exits the building.
Since the film was released in August 1939, the New Year's Eve celebration must be 31 December 1938. A theater marquee is seen, showing Irene Dunne and Charles Boyer in Love Affair (1939), which was not released until April 1939 so could not have been shown at a theater the preceding New Year's Eve.
In the credits, Irene is credited for "Miss Roger's gowns"; the apostrophe is misplaced, as it should be: Miss Rogers' gowns.
A woman trying to leave a baby at an orphanage is not proof that she is the baby's mother. At the very least, she would have to undergo a doctor's examination to determine whether she had given birth recently and to ensure that her and the baby's blood types were consistent (e.g., if Polly was type AB and the baby was type O, then it would be proven that they are not mother and daughter).
At the close of New Years Eve Niven tells Rogers that the next day they'll go for a long drive in the country. Rogers says that might be "too cold for the baby", yet she next tells him that instead they'll be in the park "all day". If a drive is too cold then the park would be too.
J. B. Merlin tells his chauffeur, "Follow my son and keep out of sight." Then Peter Merlin is shown walking down the lightly traveled street with a chauffeured, stately car slowly following him about 10 yards behind. Additionally, just to ensure Peter wouldn't be able to recognize him, J. B. covers the lower half of his face with his hand.