Julius Castle, a restaurant with a castle-like exterior located on San Francisco's Telegraph Hill, was used as the exterior of the house in the film. The filmmakers built a mansion-like exterior around parts of the restaurant to hide certain elements (such as the "Julius Castle" sign on the outside wall). Built in 1922, Julius Castle served as a high-class restaurant until it closed in 2008. (It is currently for sale.)
The grocery store where Tony worked was an actual store called Speedy's. It was open 94 years from 1914 to 2008 at 301 Union Street San Francisco, CA 94133. Google Maps Streetview shows that the corner store building still looks similar to the movie.