Robert Wise said he was willing to cast a black actor as the lead character (as it was originally written), but since there were no African-American leading actors in Hollywood at the time, he was obligated to switch the character to a white man.
Martin Scorsese is a big fan of the film and was so impressed by the boxing sequences that he had to deliberately avoid copying any of Robert Wise's camera tricks when it came his turn to make a boxing movie, Raging Bull (1980).
Based upon a narrative poem published in 1928 by Joseph Moncure March, who gave up his job as the first managing editor of "The New Yorker" to devote himself to writing. On the strength of it, he went to Hollywood as a screenwriter, remaining there for a dozen years. In 1948 he volunteered to work on this film, but was turned down. He was incensed that his black boxer Pansy Jones was changed into the white Stoker Thompson.
One of the first films to be shot using the device of real time (ie, the film lasts the same length as the events it depicts). Other notable examples of this narrative device are High Noon (1952) and Nick of Time (1995).
In the original poem, the boxer is called Pansy Jones (he goes by the name of Stoker Thompson in the film), he's black not white and, instead of being devotedly married as he is depicted in the movie, he was a bigamist. The main reason for the change of race was because RKO had no Afro-American leading men on contract at the time. James Edwards, who appears in the cast, would have fitted the bill, but was not deemed as being well-known enough to carry the movie.
Robert Wise wanted to remain faithful to the original poem The Set-Up was based on and he was keen on casting Canada Lee, a former boxer turned actor who previously appeared in Body and Soul (1947). RKO vetoed the idea since at that point a black actor never played the lead in major Hollywood film.
The trivia item below may give away important plot points.
Both The Set-Up and Champion (1949) both featured a scene in which the victorious protagonists end up being beaten by mobsters . Howard Hughes, then head of RKO which produced The Set-Up, filed a lawsuit for plagiarism against Champion's production company United Artists. Hughes won the suit and it is believed that UA retaliated by making sure The-Set Up was snubbed at the Oscars.