The only child of George Anthony (1893-1951) and Marika Palla Anthony (1913-2008), who immigrated to the United States from Greece in the 1940s, James moved at age 18 with his widowed mother to Los Angeles to pursue acting. She cleaned houses and he cleaned bathrooms to make ends meet and to pay for his acting classes.