Vermin supposedly found out that Johnny Dangerously's real last name is Kelly after Tommy gets married. However, Vermin and Johnny knew each other as kids, so Vermin would have known what his last name was already.
When assistant D.A. Tommy Kelly goes to the D.A.'s mansion to tell him he has enough evidence to convict Johnny Dangerously, he says he has, among other things, recordings, and he holds up a reel of audio tape. This scene is set in the 1930s, but magnetic tape wasn't invented until the late 1940s.
When Johnny is trying to get into the club after being dropped off by Lil, he ducks behind a white car. The poster on the cinema behind him is playing "Jack London's, Call Of The Wild" with Clarke Gable. This movie was released in 1935, where as the scene is set in the very early 30's
During the chase scene after Johnny escapes prison Alan Hale's desk sergeant character is giving updates during the chase. He is reading the information from tractor-fed paper from a dot-matrix type printer (notice the pin holes along both sides of the paper), which wasn't around in the 1930's. It wasn't widely used until the 1960's.
Before the explosion in Dundee's bathroom, the parrot is shown on the left side of the perch. In the next shot showing the parrot flapping its wings in reaction to the explosion, the parrot is on the far right side of the perch.