It is about May of 1948 in a small northern Florida town. As the film opens, Lucas Jackson (Paul Newman) is using a pipe cutter to cut the tops off of parking meters. He is drinking, possibly drunk, but not violent. When the police arrive, he is peacefully arrested. Off-screen apparently he is tried, convicted of destroying public property, and sentenced to two years in prison.
Luke and three other men have been sentenced to a rural Florida road gang performing hard labor. When he arrives at the prison, the warden reads through his record. He remarks that Luke had been in the Army, attained the rank of Sargeant, won numerous war medals, but was discharged as a private. Something he did got him busted back down to the same rank at which he entered the Army.
Dragline (George Kennedy) is clearly the leader of the prisoners. He is uneducated and illiterate, but smarter, stronger, bigger, and does not hesitate to exercise his power over them. Most of the prisoners willingly submit to Dragline, but Luke does not. There is no open opposition to Dragline, just a casual refusal to change his behavior in order to please others. Nevertheless, Dragline sees this as a threat to his power and dominance.
The men leave at sunset and perform over 12 hours daily of grueling hard labor under intense heat and humidity on the side of the road cutting grass with a sickle or shoveling dirt. One of the new prisoners passes out the first day and another one tries to get out of the detail by sweeping, which was a trick played on him. Luke makes it the first day, but exhausted and barely able to get back into the truck.
At the end of the first day for the new men, Alibi, the one that tried to get out of the hard labor by sweeping, is made to sleep in the box the first night, a cramped isolation unit outside to learn a lesson to not challenge the guards.
After Luke says something that Dragline takes as a challenge to his authority, Dragline arranges for he and Luke to have a boxing match. Being much larger, Dragline simply pounds Luke into a pulp, but Luke will not give up, or stay down on the ground. What begins as a boxing match with enthusiastic prisoners and guards watching slowly turns into a sad spectacle. Prisoners begin to plead with Luke to lay down and refuse to get back up, and eventually Dragline himself pleads with Luke to simply stop fighting back, but Luke will not stop. The prisoners begin to walk away, unable to watch the sad scene any longer. Dragline himself wants the fight to end, and at one point has to catch the beaten and exhausted Luke from falling down, carrying him across his shoulder and gently setting him on the ground, only to have Luke use what little strength he has left to tap Dragline with one last punch. Finally, even Dragline cannot continue. The only men who are still watching are the guards and the warden, but the warden seems concerned at Luke's stubbornness and the guards remain indifferent.
Luke becomes the prisoners' hero, and even Dragline is now respectful of Luke during a card game that night of having tricked a fellow opponent on betting with a bad hand dealt, giving him the name "Cool Hand Luke".
On the chain gang, Luke encourages the other prisoners, by his own attitude and energy, to excel at their menial tasks. This not only encourages camaraderie among the prisoners, it deprives the guards of a stick to hold over the prisoners' heads. The prisoners are forced to shovel sand over a freshly tarred road, and they perform the job so quickly and with such a sense of competition that they complete the job early, and thus by default earn a few hours of relaxation, because there is no more road on which to work.
On a Sunday, Luke's mother comes to visit him, visibly ill and dying of cancer or emphysema and still chain-smoking, and it is clear that Luke gets his independent streak from her. He admired her ability to live life on her own terms, and has tried to emulate that, but with much different results.
During one very hot and rainy night, Luke decides out of the blue that he can eat 50 eggs. Most of the other prisoners oppose and that it's impossible or will kill him, but Dragline begins to take bets. Luke barely succeeds at it and collects all the money from bets. Soon after, Luke receives a telegram that his mother has died.
The warden uses the death of Luke's mother as an excuse to lock him in "the box," solitary confinement in a hot wooden shed. The warden claims that men will often run away when their relatives die, so he locks Luke up until after Luke's mother is buried. This has the opposite effect, and Luke promptly escapes after being released from solitary.
Luke is later caught, after mailing a picture and magazine back to his former fellow prisoners. He is double-chained, but escapes a second time. Again, he is recaptured. While Luke is becoming the "hero" of his fellow prisoners, it is a role he does not want. He admonishes them to "stop feeding off me," to stop living life vicariously through his acts.
The guards and warden determine to break Luke's will. They force him to dig and bury, then re-dig and re-bury, a ditch. He is beaten, and tormented, until he finally begs God to spare him from the warden and guards. They take this as a sign that he is finally a broken man, and Luke himself later admits that it did break him.
A few days later, the guards have now made Luke a trustee. Nonetheless, when he is later given an opportunity, he seizes it and escapes again, stealing one of the prison trucks to make his get-away. This time, Dragline decides to go with Luke. Only later, after their escape, does Dragline weigh out the consequences. He had only two years left on his sentence, but now - if caught - will probably have many more years in prison.
Dragline wants to team up with Luke, but Luke tells him that he wants to go alone. Luke has always been an individual, not a conventional leader, and does not want to take on that role now. Luke goes into a church, after being completely irreverent to God through the entire film, and asks God to help him escape. But the warden and guards arrive, and when Luke mocks them from the church window, he is shot.
The local police want to take Luke to the hospital, but the warden insists that they take him back to prison instead. It is clear that Luke will not make it that far, but the warden heads off with Luke in his car. The one guard who always wears sunglasses is attacked by Dragline, and though Dragline is subdued, he managed to take off the guard's mirror-like sunglasses (and the warden runs over them with his car).
In the final scene, Dragline is regaling the prisoners with stories of Luke's final moments, and it is clear that Luke is dead. He is the true anti-hero, never wanting to be anybody's role model, just wanting to live life on his own terms. But like the guard's sunglasses, Luke managed to unmask the injustice and hypocricy of the system in which he was confined. Basically he gave up his life over a parking meter, but in the final analysis he could not be broken by the system.