Philip Colbert, a wealthy man from Chicago who was planning to build a factory in the rural town of Sparta, Mississippi, is found murdered. Police Chief Bill Gillespie (Rod Steiger) comes under pressure to quickly find his killer.
African-American northerner Virgil Tibbs (Sidney Poitier), passing through, is picked up at the train station between trains by a racist policeman who finds him with a substantial amount of cash in his wallet. Gillespie, heavily prejudiced against blacks, jumps to the conclusion he has his culprit but is embarrassed to learn that Tibbs is a respected Philadelphia homicide detective who had been visiting his mother. After the racist treatment he receives, Tibbs wants nothing more than to leave as quickly as possible, but his captain recommends he stay and help.
The victim's widow, already frustrated by the ineptitude of the local police, is impressed by Tibbs' expert ability when he clears another wrongly-accused suspect Gillespie has arrested on flimsy evidence. She threatens to stop construction on the much-needed factory unless he leads the investigation. Unwilling to accept help but under orders from the town's mayor, Gillespie talks Tibbs into lending his services.
Despite the rocky start to their relationship, the two policemen come to respect each other as they are forced to work together to solve the crime. Tibbs initially suspects wealthy plantation owner Eric Endicott, who opposed the new factory. When he attempts to interrogate Endicott about Colbert, Endicott slaps him in the face. Tibbs slaps him back, which leads to Endicott sending a gang of redneck hooligans after Tibbs. Gillespie rescues him from the fight and orders him to leave town for his own safety. Tibbs refuses to leave until he has solved the case.
Tibbs asks Sam Wood, the officer who discovered the body, to retrace his steps the night of the murder. He and Gillespie accompany Sam on his patrol route, stopping at a diner where the counterman, Ralph Henshaw, refuses to serve Tibbs because he is black. When Tibbs notices that Sam has deliberately changed his route, Gillespie begins to suspect Sam of the crime. When he discovers that Sam made a sizable deposit into his bank account the day after the murder (which Sam claims is gambling winnings) and Lloyd Purdy, a local, files charges against Sam for getting his 16-year-old sister, Delores, pregnant, Gillespie arrests Sam for the murder, despite Tibbs' protests. Purdy is insulted that Tibbs, a black man, was present for his sister's interrogation about her sexual encounter with Sam, and he gathers a lynch mob to get his revenge on Tibbs.
Tibbs is able to clear Sam by finding the original murder scene and pointing out that Sam would not have been able to drive two cars at the same time to dump the body and the victim's car while continuing on his patrol. Acting on a hunch, he tracks down the local back-room abortionist, who reveals that someone has paid for Delores Purdy to have an abortion. When Delores arrives, Tibbs pursues her outside, where he is confronted by the murderer... whom is the diner counterman Ralph Henshaw. Purdy's mob tracks down Tibbs at this moment, and he is being held at gunpoint when he proves to Purdy that it was Ralph, not Sam, who got Delores pregnant. Purdy attacks Ralph, who kills Purdy in self-defense. Ralph is arrested and confesses to the murder of Colbert. He had attempted to rob Colbert to gain money to pay for Delores' abortion but accidentally killed him.
His job done, Tibbs finally boards the Gulf, Mobile & Ohio train out of town, seen off by a more respectful Gillespie.