Turmoil grips Shanghai in the 1940s. Various gangs vie for power, the most feared of which is The Axe Gang, led by the infamous Brother Sum and aptly named after its weapon of choice. In the absence of law enforcement, people can live peacefully only in poor areas which do not appeal to the gangs. An example is Pig Sty Alley, a tenement home to people of various trades, run by a lecherous landlord and his domineering wife. One day, two troublemakers, Sing and Bone, come to the alley impersonating members of the Axe Gang in order to gain respect. Their plan fails miserably, and Sing's antics attract the real gang to the scene. In the massive brawl that ensues, more than fifty gangsters are defeated by three tenants who are actually powerful martial arts masters: Coolie, Master of the "Twelve Kicks;" Tailor, master of the "Iron Fist;" and Donut, master of the "Hexagon Staff."
After the fight, Sing and Bone are apprehended by Brother Sum for causing trouble and publicly humiliating the Axe Gang. The two narrowly escape death when Sing quickly picks the locks on the chains with which they are bound. Sing asks Sum to make himself and Bone members of the Axe Gang. Impressed with his skill at lockpicking, Sum tells them that if they kill just one person, he will allow them to join the gang. The next day the duo return to Pig Sty Alley to murder the Landlady, but fail comically due to ineptitude. The two part ways and narrowly escape from the furious Landlady. Sing is badly injured and hides in a traffic control pulpit, where his injuries spontaneously heal. During this convalescence he unconsciously strikes the steel sides of the pulpit, leaving deep impressions of his hands. After he has fully recovered, he rejoins Bone, but is unable to explain his mysterious healing.
Sing and Bone lament their failure on the streets, and Sing describes his childhood. He spent his meagre life savings to buy a Buddhist Palm manual from a beggar with the intention of "preserving world peace." He trained himself, but when he tried to defend a mute girl from bullies trying to steal her lollipop, he was beaten. Sing concludes that good guys never win and decides to become a criminal. After telling his story, the duo then steal ice cream from a street vendor, laughing maniacally as they escape from her on a tram.
Angered by his gang's defeat, Brother Sum hires the Harpists, a pair of skilled assassins who fight using a magical Guqin. They strike Pig Sty Alley at night as Coolie, Tailor and Donut are preparing to depart, having been evicted for antagonizing the gang. The three are quickly overwhelmed, prompting the Landlord and Landlady, revealed to be Kung fu masters of even greater prowess, to intervene. Although the killers and the Axe Gang are driven off, the three evictees suffer fatal injuries: Coolie is decapitated, Tailor dies of stab wounds, and Donut dies of blunt-force trauma. The Landlord and Landlady evacuate Pig Sty Alley out of concern for their tenants' safety.
The following day, after being humiliated and pummeled by a clerk he had previously antagonized, Sing mugs the female ice cream vendor from the earlier scene. She is revealed to be the mute girl from his childhood whom he had vainly tried to defend. He recognizes the old lollipop she offers him as a token of her gratitude, but becomes upset, rebuffing her and berating Bone. In exasperation, he drives Bone away and is picked up by members of the Axe Gang. Brother Sum, having earlier witnessed Sing's ability to quickly pick locks, instructs him to sneak into a mental asylum to free The Beast, a mercenary rumored to be the most dangerous person alive.
The Beast is freed and brought over to the Axe Gang headquarters behind a casino. However, his flippant attitude and sloppy appearance puzzles Sum and his men. Soon, they are ultimately convinced when The Beast fires a gun point-blank at his own head then stops the bullet between two fingers. The Beast, however, loses interest in Sum, and approaches the Landlord and Landlady, who have come to Sum's casino to deal with him and his gang. The three masters engage one another, and although initially overcome, the Landlady is able to subdue The Beast by breaking the top off a giant funeral bell and using it as a megaphone to amplify her sonic technique, the Lion's Roar. The Beast, injured, pretends to surrender, but then attempts to stab his opponents with poisoned barbs. Although he fails, the Landlord and Landlady are forced to lock their bodies with his to counter his attack, and all three become unable to move. Sum orders Sing to kill the Landlady and Landlord, but feeling reformed, Sing beats away Sum and hits The Beast instead. Enraged, The Beast breaks free of the joint-lock and pummels Sing. Fortunately, Sing is spirited away by the Landlord and Landlady when The Beast's back is turned. The Beast, annoyed by his complaints, casually kills Sum, and gives chase.
At Pig Sty Alley, Sing, while wrapped head-to-toe in bandages and treated with Chinese medicine, undergoes a metamorphosis. He quickly recovers from his wounds, and his latent potential as a "natural-born Kung Fu genius" is realized. He engages the Axe Gang and The Beast, fending off the gangsters with ease. However The Beast puts up a fight, and his Toad Technique sends Sing rocketing into the sky. As he falls back to Earth, Sing recalls the Buddhist Palm, and delivers a blow that knocks The Beast flat and leaves a huge, hand-shaped crater in the ground. The Beast again pretends to surrender, but when Sing relents, The Beast tries to stab him with his poisoned barb, just as he had tried to stab the Landlord and Landlady. Sing calmly performs another Buddhist Palm technique, leaving a three-story hole in the wall of the tenement, although he intentionally avoids hitting The Beast. Stunned, The Beast remains immobile while Sing relieves him of his weapon. The Beast, stammering, asks Sing what sort of technique he had just performed - Sing offers to teach him, and, weeping, The Beast kneels at his feet and admits the battle is over, Sing victorious.
Some time later, Sing and Bone open a candy store. When the mute ice cream vendor walks by, Sing goes out to meet her. The two see each other as their childhood selves, and run happily into the shop while some people of Pig Sty Alley, including the Landlord and Landlady are shown as normal citizens. Outside, the same beggar who sold Sing the Buddhist Palm manual offers a selection of martial arts manuals to a boy eating a lollipop.