Thomas Levy (Dustin Hoffman), nicknamed "Babe" in the film, is a history Ph.D. candidate researching the same field as his father, who committed suicide after being investigated during the Joseph McCarthy era. Babe is also an avid runner, which lends the film its title. Babe's brother, Henry (Roy Scheider), better known as "Doc", poses as an oil company executive but in fact is a U.S. government agent working for a secret agency headed up by Director Peter Janeway (William Devane). Babe is not aware of his brother's true occupation.
Doc is often supposedly out of the country on business for extended periods of time but comes to New York under the guise of a visit to Babe. The brother of a Nazi war criminal possesses a safety deposit box key, but is killed in a traffic accident (after a road rage altercation with a short tempered middle-aged Jewish American motorist). Doc suspects that the criminal, Dr. Christian Szell (Laurence Olivier) will be arriving to retrieve an extremely valuable diamond collection. Szell later sends a burly assassin named Chen (James Wing Woo) to kill Doc in his Paris hotel room, but Doc fights the hit man and breaks his back.
Babe enters into a relationship with a young woman named Elsa Opel (Marthe Keller), who claims to be from Switzerland. One night while out on a date Elsa and Babe are mugged in a park by two men dressed in suits. Some time later, Doc takes the couple to lunch, where he tricks Elsa into revealing that she has been lying to Babe about her background. Though Doc suspects she may have some connection to Szell, he tells Babe only that she is seeking an American husband so that she can become a U.S. citizen.
After Szell arrives in America, Doc meets him to tell the former Nazi he is not welcome in the country, and also to warn Szell to stay away from Babe. Szell casually accepts the pronouncement, but then swiftly knifes Doc, wounding him severely. Doc is able to make it back to his brother's apartment, but collapses and dies in Babe's arms without telling him anything. The police interrogate Babe for hours, until government agents led by Peter Janeway arrive. Janeway asks Babe what Doc told him before he died, and informs Babe of his brother's career as a U.S. government agent. Babe insists that his brother did not tell him anything, but Janeway feels that Doc struggled all the way to Babe's apartment to give him vital information of some kind.
Babe is later abducted from his apartment by Szell's subordinates, Karl and Erhardt (the two men from the park). In an infamous sequence, Babe is tortured by Szell, a skilled dentist, who repeatedly asks "Is it safe?" Confused by the question code phrase he does not understand, Babe denies any knowledge, but is tortured. The dentist offers him oil of cloves, an anesthetic, as positive inducement to cooperate. Eventually, Babe loses consciousness and Szell pauses his torture.
Babe is then rescued by Janeway, who apparently kills Szell's bodyguards and takes Babe from Szell's hideout. As he drives, Janeway explains that Szell is in America to sell off his large cache of diamonds, which he had taken from Jews he had exterminated at Auschwitz during World War II. Janeway continually presses Babe about Doc's dying words, but Babe again insists he knows nothing. Frustrated, Janeway reveals himself as a double agent working with the Nazi criminal all along, and he turns Babe over to Karl and Erhardt (Janeway had only faked killing them). Szell, it turns out, is one of Janeway's highest level informants, and had informed on other Nazi war criminals in return for immunity.
Delivered back into Szell's hands, the Nazi has a curiously kindly conversation with Babe before calmly explaining why he is holding Babe for questioning. Szell suspected that Doc would attempt to rob him of his diamonds, or rat on him to authorities; thus his desire to know if it's "safe" to withdraw the diamonds, i.e., whether Doc told Babe any incriminating information. Still unable to extract anything from Babe, he proceeds to drill into one of Babe's healthy teeth. Babe eventually escapes again, this time on his own, with Janeway and Szell's two henchmen giving chase in a car. Though exhausted and barefoot, he is able to outrun his pursuers in part by remembering famed "marathon man" Abebe Bikila, who ran barefoot.
After inviting a neighborhood acquaintance and his toughs to break into his apartment and steal his pistol (the same pistol his father used to commit suicide), Babe phones Elsa, who agrees to meet him with a car. She drives him to a country home as a hideout. Babe guesses that she has set him up; she confesses that Szell's dead brother owned the home, and that she was one of Szell's couriers. Janeway and Szell's men arrive, but Babe avoids an ambush by taking Elsa hostage. In another twist, Janeway kills Szell's men and offers to let Babe kill Szell in revenge for Doc's death if Janeway can have the diamonds. Babe agrees, but as he leaves to find Szell, Janeway attempts to shoot Babe, but kills Elsa instead when she tries to alert Babe. Angered, Babe guns down Janeway.
Back in New York, Szell attempts to determine the value of his diamonds. However, he chooses an appraiser in the Diamond District in midtown Manhattan, where many of the shop owners are Jewish. A shop assistant who is also an Auschwitz survivor believes he has recognized Szell as a wanted Nazi criminal. After Szell hurriedly leaves the shop, an elderly Jewish woman also recognizes him, but passersby think she is senile. Trying to cross the street to get closer to Szell in order to expose him she is hit by a taxi, causing a crowd to assemble to aid her. Amidst the confusion, the shop assistant appears again, directly confronting Szell. Becoming increasingly paranoid, Szell slits the man's throat with a retractable blade concealed in his sleeve.
Szell retrieves his diamonds from the bank, but is taken hostage by Babe as he attempts to leave, who inconspicuously forces him into Central Park and into one of the pump rooms at the south end of the Reservoir. Babe holds Szell at gunpoint on a scaffold and seizes the diamonds; rather than kill Szell, Babe informs him that he will allow Szell to live and keep as many diamonds as he can swallow. Szell initially refuses, prompting Babe to begin throwing the diamonds into the water below them. Szell relents and swallows one diamond, but then refuses to cooperate further. Szell goes on to insult Babe, and after he spits in Babe's face a fight erupts in which Szell tries to stab Babe. Babe throws the remainder of the diamonds down the scaffold steps towards the water; Szell dives for them, but stumbles, and fatally falls on his own knife blade. Picking up his gun, Babe exits the pump room and heads out into Central Park. Stopping by the Reservoir, he throws the gun into the water.