The film opens with a series of still photographs appearing over melancholic music, representing the abstract memories of the unseen Herman Raucher, now a middle-aged man. We then hear Raucher (voiced by Robert Mulligan) recalling the summer he spent on the island of Nantucket in 1942. The film flashes back to a day that then 15-year-old Herman "Hermie" Raucher (Gary Grimes) and his friends jock Oscy (Jerry Houser) and introverted nerd Benjie (Oliver Conant) spent playing on the beach. They are the self-proclaimed 'Terrible Trio'. They spot a young soldier carrying his new bride (Jennifer O'Neill) into a house on the beach and are struck by her beauty, especially Hermie, who is unable to get her out of his mind.
They continue spending afternoons on the beach where, in the midst of scantily-clad teenage girls, their thoughts invariably turn to sex. All of them are virgins: Oscy is obsessed with the act of sex, while Hermie finds himself developing romantic interest in the bride, whose husband he spots leaving the island on a water taxi one morning. Later that day, Hermie finds her trying to carry bags of groceries by herself, and helps get them back to her house. They strike up a friendship and he agrees to return to help her with chores.
Meanwhile, Oscy and Hermie, thanks to a sex manual discovered by Benjie, become convinced they know everything necessary to lose their virginity. Led by Oscy, they test this by going to the cinema and picking-up a trio of high-school girls. Oscy stakes out the most attractive one, Miriam (Christopher Norris), "giving" Hermie her less attractive friend, Aggie (Katherine Allentuck) and leaving Benjie with Gloria, a heavyset girl with braces. Frightened by the immediacy of sex, Benjie runs off, and is not seen by Hermie or Oscy again that night.
Hermie and Oscy spend the entirety of the evening's film attempting to "put the moves" on Miriam and Aggie. Oscy pursues Miriam, eventually making out with her during the movie, and later learns her ways are well-known on the island. Hermie finds himself succeeding with Aggie, who allows him to grope what he thinks is her breast; Oscy later points out Hermie was fondling her arm.
The next morning, Hermie helps the bride move boxes into her attic and she thanks him by giving him a kiss on the forehead. Later, in preparation for a marshmallow roast on the beach with Aggie and Miriam, Hermie goes to the local drugstore. In a painfully humorous sequence he builds up the nerve to ask for condoms.
That night, Hermie roasts marshmallows at the beach with Aggie while Oscy succeeds in having sex with Miriam between the dunes. He is so successful he sneaks over to Hermie and Aggie to ask for more condoms. Confused as to what's happening, Aggie follows Oscy back, where she sees him having sex with Miriam and runs home, upset.
The next day, Hermie comes across the bride sitting outside her house, writing to her husband. Hermie offers to keep her company that night and she says she looks forward to seeing him, revealing her name is Dorothy. She finally tells Hermie a little about herself; she is 22-years-old, and has been married to her husband for just under a year until he was recently drafted and has been sent to England in the US Army Air Corps. An elated Hermie goes home and puts on a suit, dress shirt and heads back to Dorothy's house, running into Oscy on the way; Oscy relates that Miriam's appendix burst and she's been rushed to the mainland. Hermie, convinced he is at the brink of adulthood because of his relationship with Dorothy, brushes Oscy off.
He heads to her house, which is eerily quiet. Going in, he discovers a bottle of whiskey, several cigarette butts in an ashtray, and a telegram from the government. Dorothy's husband is dead; his plane shot down over France. Dorothy comes out of her bedroom, crying, and Hermie tells her "I'm sorry." The sense of empathy triggers her to channel to Hermie some of her loneliness. She turns on the record player and invites Hermie to dance with her. They kiss and embrace, tears on both their faces. Without speaking, and to the sound only of the waves, they move to the bedroom, where she draws him into bed and gently makes love with him.
Afterward, withdrawing again into her world of hurt, Dorothy retires to the porch, leaving Hermie alone in her bedroom. He approaches her on the porch, where she can only quietly say "Good night, Hermie." He leaves, his last image of Dorothy being of her leaning against the railing, as she smokes a cigarette and stares into the night sky.
At dawn, Hermie meets Oscy and the two share a moment of reconciliation, with Oscy informing Hermie that Miriam will recover. Oscy, in an uncharacteristic act of sensitivity, lets Hermie be by himself, departing with the words, "Sometimes life is one big pain in the ass."
Trying to sort out what has happened, Hermie goes back to Dorothy's house. Dorothy has fled the island in the middle of the night and an envelope is tacked to the front door with Hermie's name on it. Inside is a note from Dorothy, saying she hopes he understands she must go back home as there is much to do to sort out her late husband's business. She assures Hermie she will never forget him, and he will find his way of remembering what happened that night. Her note closes with the hope that Hermie may be spared the senseless tragedies of life.
In the final scene, Hermie, suddenly approaching manhood, is seen looking at Dorothy's old house and the ocean from a distance before he turns to join his friends. To bittersweet music, the adult Raucher sadly recounts that he never saw Dorothy again or learned what became of her.
(After the film ends, the cast is given by showing a still photo of each of the primary cast members with a superimposed name. The movie had exactly eight on-screen speaking characters: Dorothy, the Terrible Trio, the three teenage girls and the druggist.)