In the opening scene, private detective Sam Grunion (Groucho Marx) explains to the viewers that he has been searching for the extremely valuable Royal Romanoff diamonds for eleven years, and his investigation leads him to a troupe of struggling performers, led by Mike Johnson (Paul Valentine), who are trying to put on a musical revue called 'Love Happy'.
Grunion notes that the impoverished young dancers would starve were it not for the sweet, silent Harpo (Harpo Marx), at Herbert & Herbert, a gourmet food shop that also trafficks in stolen diamonds. Harpo kindly helps ladies with their shopping bags, all the while pilfering their groceries and stuffing them in the pockets of his long trench coat. When the elegant Madame Egelichi (Ilona Massey) arrives, store manager Lefty Throckmorton (Melville Cooper) tells her that "the sardines" have come in. Harpo sneaks into the basement and watches as Lefty lovingly unpacks a sardine can marked with a Maltese cross, and swipes the can from Lefty's pocket, replacing it with an unmarked one. Madame Egelichi, who has gone through eight husbands in three months in her quest for the Romanoff diamonds, is furious when Lefty produces the wrong can. When Lefty remembers seeing Harpo in the basement, she orders him to call the police and offer a $1,000 reward for his capture.
At the theater, meanwhile, unemployed entertainer Faustino the Great (Chico Marx) asks Mike for a job as a mind-reader, and when Faustino's clever improvisation stops the show's backer, Mr. Lyons (Leon Belasco), from repossessing the scenery, Mike gratefully hires him. Harpo, who is secretly in love with dancer Maggie Phillips (Vera-Ellen), Mike's girl friend, gives her the sardine can, and she says she will eat them tomorrow. A policeman sees Harpo inside the theater and brings him to Madame Egelichi, who turns Harpo over to her henchmen, Alphonse (Raymond Burr) and Hannibal (Bruce Gordon) Zoto. After three days of interrogation, Harpo still refuses to talk, and when he is left alone, he calls Faustino at the theater, using the bike horn he carries in his pocket to communicate. Madame Egelichi listens on the extension as Faustino declares that there are plenty of sardines at the theater, and she goes there at once.
Meanwhile, Mike has just finished telling the troupe that they do not have enough money to open when Madame Egelichi arrives and offers to finance the show. Mike cancels his plans to take Maggie out for her birthday so that he and his new backer can discuss the arrangements. In the alley outside the theater, Harpo, having escaped from Madame Egelichi's suite, finds the diamonds in the sardine can which had been set out for a cat, and puts them in his pocket. When he finds Maggie crying in her dressing room, Harpo takes her to Central Park, where he plays the harp for her and gives her the diamonds as a birthday gift.
On the opening night of the show, Grunion is visited by an agent of the Romanoff family, who threatens to kill him if he does not produce the diamonds in an hour. At the theater, Lefty and the Zoto brothers spy through a window as Maggie puts on the diamond necklace, but Mike asks her not to wear it, promising to buy her an engagement ring instead. As they kiss, Maggie removes the necklace and drops it on the piano strings. The curtain goes up, and when Harpo sees Lefty and the Zoto brothers menacing Maggie, he distracts them with a piece of costume jewelry and leads them up to the roof.
Meanwhile, on stage, Faustino plays the piano, and when he strikes the keys forcefully, the diamond necklace flies into the air, drawing the attention of Madame Egelichi, who is watching from the audience. Faustino pockets the diamonds, then rushes to the roof to help Harpo. Madame Egelichi shows up with a gun and demands the necklace, but Faustino gives her the fake diamonds. After tying up Lefty and the Zotos and recovering the real diamonds, Harpo encounters Grunion, who has been hiding on the roof. Harpo drops the diamonds in Grunion's pocket, but then steals them back as Madame Egelichi begins to lead the detective away.
In the final scene, back in his office, Grunion comments that Harpo disappeared with the diamonds, never realizing their true value. Grunion interrupts his story to take a phone call from his wife, who turns out to be the former Madame Egelichi.