Harry Caul (Gene Hackman), a private surveillance expert in San Francisco, has been hired by the Director to follow and record the actions of his wife, Ann (Cindy Williams), and her lover, Marc (Frederic Forrest). Using three separate microphones, Caul and his associates follow the couple around a park. Later, he pieces together a conversation that clearly indicates the two are in a relationship and that they fear being found out by the Director.
Caul is a secretive, almost paranoid individual. He alienates his assistant, Stan (John Cazale), because he won't teach him more about audio surveillance, and his girlfriend, Amy (Teri Garr), because he won't share with her details about his career or personal life. He also refuses to hand over the tapes of Ann and Marc to the Director's assistant, Martin (Harrison Ford). The Director had specified that the tapes were to be turned over only to him. Martin protests but Caul keeps the tapes.
Caul becomes increasingly concerned that, were he to turn the tapes over to the Director, that some harm would come to Ann and Marc. On the tapes, Marc can be heard to say "He'd kill us if he had the chance." When he worked on the East Coast, Caul had participated in a surveillance that led to the deaths of two persons. He worries that this might happen again, and goes to church to confess his sins.
After Amy breaks off their relationship, Caul attends a surveillance convention. Martin also attends, and Caul confronts him and orders Martin to stop following him. He learns that Stan has gone to work for Moran (Alan Garfield), a Detroit-based surveillance expert looking to expand to the West Coast. Moran demonstrates to convention goers a new remote audio system that turns any telephone into a live microphone. Although Caul persuades Stan not to work for Moran after the convention is over, Moran is effusive in his praise for Caul. Lonely and enjoying the positive attention, Caul lets his guard down and takes Moran, Stan and several others back to his workroom for a post-convention party. Among the partygoers is Meredith (Elizabeth McRae), a showgirl who worked at Moran's convention booth. Meredith openly flirts with Caul and in a private moment, Caul opens up to her about his troubles with Amy. This conversation is recorded by Moran, who had earlier planted a bug on Caul. Caul is outraged and kicks everybody out. Meredith stays, helps Caul into bed, undresses herself and sleeps with him. Caul has a dream in which he pursues Ann through the fog, telling her about how he nearly died as a child. In this dream, he warns Ann that the Director plans to kill her. "I'm not afraid of death," he tells her as she disappears into the mist, "but I am afraid of murder."
When Caul awakens in the morning, he discovers both Meredith and the tapes are gone. Frustrated and angry, he returns home and tries to call the Director. Caul has lied to many people that he doesn't have a home telephone and is surprised when Martin calls him back at home. Martin tells him that he took the tapes and delivered them to the Director. Caul should come to the office to pick up his fee ($15,000). Caul does as asked, and finds the seething Director (Robert Duvall) listening to the conversation. The Director curtly dismisses Caul after paying him.
Disturbed, Caul goes to the hotel that Ann and Marc mentioned as meeting place in their conversation. He rents an adjoining room and inserts a microphone through the wall to listen to the conversation. He hears shouting. Rushing to the balcony, he sees bloody hand prints on the glass partition. Appalled and frightened, he collapses in his own room. Upon reviving, he breaks into the room Marc and Ann had rented. The room seems orderly, but the toilet is running. He gives it a flush and blood wells out.
Caul goes to see the Director but security blocks his way. On the street, he is surprised to see Ann sitting in an idling car. A newspaper headline informs him that the Director has been killed in a car accident. Ann now controls the Director's company. Later, as Ann, Marc and Martin try to escape reporters asking them about an inquiry into the Director's death, they notice Caul watching them. Caul realizes that the bloody hand he saw in the hotel was the Director's; Marc's statement that the Director would kill them if he had the chance was a rationalization of their decision to kill him.
Back in his apartment, Caul sits alone, playing his saxophone to his favorite jazz records. His phone rings. When he answers, no one is there. He hangs up and resumes playing. It rings again; this time, Martin tells him that Ann and Marc know he has deduced what happened, and that they will be listening to him. Martin then plays back a recording of Caul's saxophone playing. Completely consumed by paranoia and forgetful of Moran's demonstration at the convention, Caul demolishes his apartment looking for the bug. Unable to find it, he resumes playing his saxophone.