Cross-evaluation of metrics to estimate the significance of creative works
by dan_dassow | created - 30 Aug 2015 | updated - 09 Sep 2015 | PublicFull Article: http://www.pnas.org/content/112/5/1281.full
Supporting Information: http://www.pnas.org/content/suppl/2015/01/15/1412198112.DCSupplemental/pnas.201412198SI.pdf#nameddest=ST3
vol. 112 no. 5 > Max Wasserman, 1281–1286, doi: 10.1073/pnas.1412198112
Authors: Max Wasserman, Xiao Han T. Zeng, and Luís A. Nunes Amaral
Edited by Kenneth W. Wachter, University of California, Berkeley, CA, and approved December 1, 2014 (received for review June 27, 2014)
Significance
Whether it is Hollywood movies or research papers, identifying works of great significance is imperative in a modern society overflowing with information. Through analysis of a network constructed from citations between films as referenced in the Internet Movie Database, we obtain several automated metrics for significance. We find that the best automated method can identify significant films, represented by selection to the US National Film Registry, at least as well as the aggregate rating of many experts and far better than the rating of a single expert. We hypothesize that these results may hold for other creative works.
Abstract In a world overflowing with creative works, it is useful to be able to filter out the unimportant works so that the significant ones can be identified and thereby absorbed. An automated method could provide an objective approach for evaluating the significance of works on a universal scale. However, there have been few attempts at creating such a measure, and there are few “ground truths” for validating the effectiveness of potential metrics for significance. For movies, the US Library of Congress’s National Film Registry (NFR) contains American films that are “culturally, historically, or aesthetically significant” as chosen through a careful evaluation and deliberation process. By analyzing a network of citations between 15,425 United States-produced films procured from the Internet Movie Database (IMDb), we obtain several automated metrics for significance. The best of these metrics is able to indicate a film’s presence in the NFR at least as well or better than metrics based on aggregated expert opinions or large population surveys. Importantly, automated metrics can easily be applied to older films for which no other rating may be available. Our results may have implications for the evaluation of other creative works such as scientific research.
Key Words: data science complex networks citations films IMDb
List based on: Table S3. Films with most long-gap citations
LGC = *Long-gap citation count.
NFR year = Year inducted into the US Library of Congress’s National Film Registry (NFR)
- Instant Watch Options
- Genres
- Movies or TV
- IMDb Rating
- In Theaters
- Release Year
- Keywords
1. The Wizard of Oz (1939)
G | 102 min | Adventure, Family, Fantasy
Dorothy Gale is swept away from a farm in Kansas to a magical land of Oz in a tornado and embarks on a quest with her new friends to see the Wizard who can help her return home to Kansas and help her friends as well.
Directors: Victor Fleming, George Cukor, Mervyn LeRoy, Norman Taurog, Richard Thorpe, King Vidor | Stars: Judy Garland, Frank Morgan, Ray Bolger, Bert Lahr
Votes: 365,074 | Gross: $2.08M
Year LGC NFR year 1939 565 1989
2. Star Wars (1977)
14 | 121 min | Action, Adventure, Fantasy
Luke Skywalker joins forces with a Jedi Knight, a cocky pilot, a Wookiee and two droids to save the galaxy from the Empire's world-destroying battle station, while also attempting to rescue Princess Leia from the mysterious Darth Vader.
Director: George Lucas | Stars: Mark Hamill, Harrison Ford, Carrie Fisher, Alec Guinness
Votes: 1,197,850 | Gross: $322.74M
Year LGC NFR year 1977 297 1989
3. Psycho (1960)
18A | 109 min | Horror, Mystery, Thriller
A Phoenix secretary embezzles forty thousand dollars from her employer's client, goes on the run, and checks into a remote motel run by a young man under the domination of his mother.
Director: Alfred Hitchcock | Stars: Anthony Perkins, Janet Leigh, Vera Miles, John Gavin
Votes: 581,561 | Gross: $32.00M
Year LGC NFR year 1960 241 1992
4. Casablanca (1942)
G | 102 min | Drama, Romance, War
A cynical American expatriate struggles to decide whether or not he should help his former lover and her fugitive husband escape French Morocco.
Director: Michael Curtiz | Stars: Humphrey Bogart, Ingrid Bergman, Paul Henreid, Claude Rains
Votes: 506,607 | Gross: $1.02M
Year LGC NFR year 1942 212 1989
5. Gone with the Wind (1939)
G | 238 min | Drama, History, Romance
A manipulative woman and a roguish man conduct a turbulent romance during the American Civil War and Reconstruction periods.
Directors: Victor Fleming, George Cukor, Sam Wood | Stars: Clark Gable, Vivien Leigh, Thomas Mitchell, Barbara O'Neil
Votes: 281,669 | Gross: $198.68M
Year LGC NFR year 1939 198 1989
6. King Kong (1933)
PG | 100 min | Adventure, Horror, Sci-Fi
A film crew goes to a tropical island for an exotic location shoot and discovers a colossal ape who takes a shine to their female blonde star. He is then captured and brought back to New York City for public exhibition.
Directors: Merian C. Cooper, Ernest B. Schoedsack | Stars: Fay Wray, Robert Armstrong, Bruce Cabot, Frank Reicher
Votes: 77,136 | Gross: $10.00M
Year LGC NFR year 1933 191 1991
7. Frankenstein (1931)
(Banned) | 70 min | Drama, Horror, Sci-Fi
An obsessed scientist assembles a living being from parts of exhumed corpses.
Director: James Whale | Stars: Colin Clive, Mae Clarke, Boris Karloff, John Boles
Votes: 62,887
Year LGC NFR year 1931 170 1991
8. The Godfather (1972)
18A | 175 min | Crime, Drama
The aging patriarch of an organized crime dynasty transfers control of his clandestine empire to his reluctant son.
Director: Francis Ford Coppola | Stars: Marlon Brando, Al Pacino, James Caan, Diane Keaton
Votes: 1,559,993 | Gross: $134.97M
Year LGC NFR year 1972 162 1990
9. Citizen Kane (1941)
G | 119 min | Drama, Mystery
Following the death of publishing tycoon Charles Foster Kane, reporters scramble to uncover the meaning of his final utterance; 'Rosebud'.
Director: Orson Welles | Stars: Orson Welles, Joseph Cotten, Dorothy Comingore, Agnes Moorehead
Votes: 386,539 | Gross: $1.59M
Year LGC NFR year 1941 143 1989
10. 2001: A Space Odyssey (1968)
G | 149 min | Adventure, Sci-Fi
After discovering a mysterious artifact buried beneath the Lunar surface, mankind sets off on a quest to find its origins with help from intelligent supercomputer H.A.L. 9000.
Director: Stanley Kubrick | Stars: Keir Dullea, Gary Lockwood, William Sylvester, Daniel Richter
Votes: 583,212 | Gross: $56.95M
Year LGC NFR year 1968 143 1991
11. Jaws (1975)
R | 124 min | Adventure, Thriller
When a killer shark unleashes chaos on a beach community, it's up to a local sheriff, a marine biologist, and an old seafarer to hunt the beast down.
Director: Steven Spielberg | Stars: Roy Scheider, Robert Shaw, Richard Dreyfuss, Lorraine Gary
Votes: 532,457 | Gross: $260.00M
Year LGC NFR year 1975 129 2001
12. Night of the Living Dead (1968)
R | 96 min | Horror
A ragtag group of Pennsylvanians barricade themselves in an old farmhouse to remain safe from a bloodthirsty, flesh-eating breed of monsters who are ravaging the East Coast of the United States.
Director: George A. Romero | Stars: Duane Jones, Judith O'Dea, Karl Hardman, Marilyn Eastman
Votes: 112,414 | Gross: $0.09M
Year LGC NFR year 1968 122 1999
13. It's a Wonderful Life (1946)
G | 130 min | Drama, Family, Fantasy
An angel is sent from Heaven to help a desperately frustrated businessman by showing him what life would have been like if he had never existed.
Director: Frank Capra | Stars: James Stewart, Donna Reed, Lionel Barrymore, Thomas Mitchell
Votes: 385,708
Year LGC NFR year 1946 109 1990
14. The Graduate (1967)
14A | 106 min | Comedy, Drama, Romance
A disillusioned college graduate finds himself torn between his older lover and her daughter.
Director: Mike Nichols | Stars: Dustin Hoffman, Anne Bancroft, Katharine Ross, William Daniels
Votes: 248,634 | Gross: $104.95M
Year LGC NFR year 1967 97 1996
15. Vertigo (1958)
G | 128 min | Mystery, Romance, Thriller
A former police detective juggles wrestling with his personal demons and becoming obsessed with a hauntingly beautiful woman.
Director: Alfred Hitchcock | Stars: James Stewart, Kim Novak, Barbara Bel Geddes, Tom Helmore
Votes: 348,930 | Gross: $3.20M
Year LGC NFR year 1958 92 1989
16. Snow White and the Seven Dwarfs (1937)
G | 83 min | Animation, Family, Fantasy
Exiled into the dangerous forest by her wicked stepmother, a princess is rescued by seven dwarf miners who make her part of their household.
Directors: William Cottrell, David Hand, Wilfred Jackson, Larry Morey, Perce Pearce, Ben Sharpsteen | Stars: Adriana Caselotti, Harry Stockwell, Lucille La Verne, Roy Atwell
Votes: 176,080 | Gross: $184.93M
Year LGC NFR year 1937 91 1989
17. Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb (1964)
PG | 95 min | Comedy
An insane general triggers a path to nuclear holocaust that a War Room full of politicians and generals frantically tries to stop.
Director: Stanley Kubrick | Stars: Peter Sellers, George C. Scott, Sterling Hayden, Keenan Wynn
Votes: 438,335 | Gross: $0.28M
Year LGC NFR year 1964 91 1989
18. Dracula (1931)
PG | 75 min | Drama, Fantasy, Horror
The ancient vampire Count Dracula arrives in England and begins to prey upon the virtuous young Mina.
Directors: Tod Browning, Karl Freund | Stars: Bela Lugosi, Helen Chandler, David Manners, Dwight Frye
Votes: 44,714
Year LGC NFR year 1931 90 2000
19. The Maltese Falcon (1941)
PG | 100 min | Film-Noir, Mystery
A private detective takes on a case that involves him with three eccentric criminals, a gorgeous liar, and their quest for a priceless statuette.
Director: John Huston | Stars: Humphrey Bogart, Mary Astor, Gladys George, Peter Lorre
Votes: 146,101 | Gross: $2.11M
Year LGC NFR year 1941 80 1989
20. Bambi (1942)
G | 70 min | Animation, Drama, Family
The story of a young deer growing up in the forest.
Directors: James Algar, Samuel Armstrong, David Hand, Graham Heid, Bill Roberts, Paul Satterfield, Norman Wright, Arthur Davis, Clyde Geronimi | Stars: Hardie Albright, Stan Alexander, Bobette Audrey, Peter Behn
Votes: 127,433 | Gross: $102.80M
Year LGC NFR year 1942 79 2011
21. The Exorcist (1973)
R | 122 min | Horror
When a 12 year-old girl is possessed by a mysterious entity, her mother seeks the help of two priests to save her.
Director: William Friedkin | Stars: Ellen Burstyn, Max von Sydow, Linda Blair, Lee J. Cobb
Votes: 352,286 | Gross: $232.91M
Year LGC NFR year 1973 78 2010
22. Taxi Driver (1976)
18A | 114 min | Crime, Drama
A mentally unstable veteran works as a nighttime taxi driver in New York City, where the perceived decadence and sleaze fuels his urge for violent action by attempting to liberate a presidential campaign worker and an underage prostitute.
Director: Martin Scorsese | Stars: Robert De Niro, Jodie Foster, Cybill Shepherd, Albert Brooks
Votes: 697,301 | Gross: $28.26M
Year LGC NFR year 1976 71 1994
23. Sunset Blvd. (1950)
PG | 110 min | Drama, Film-Noir
A screenwriter develops a dangerous relationship with a faded film star determined to make a triumphant return.
Director: Billy Wilder | Stars: William Holden, Gloria Swanson, Erich von Stroheim, Nancy Olson
Votes: 194,021
Year LGC NFR year 1950 70 1989
24. Planet of the Apes (1968)
G | 112 min | Adventure, Sci-Fi
An astronaut crew crash-lands on a planet in the distant future where intelligent talking apes are the dominant species, and humans are the oppressed and enslaved.
Director: Franklin J. Schaffner | Stars: Charlton Heston, Roddy McDowall, Kim Hunter, Maurice Evans
Votes: 162,149 | Gross: $33.40M
Year LGC NFR year 1968 69 2001
25. Deliverance (1972)
18A | 109 min | Adventure, Drama, Thriller
Intent on seeing the Cahulawassee River before it's dammed and turned into a lake, outdoor fanatic Lewis Medlock takes his friends on a canoeing trip they'll never forget into the dangerous American back-country.
Director: John Boorman | Stars: Jon Voight, Burt Reynolds, Ned Beatty, Ronny Cox
Votes: 96,076 | Gross: $7.06M
Year LGC NFR year 1972 66 2008
26. The Sound of Music (1965)
G | 172 min | Biography, Drama, Family
A woman leaves an Austrian convent to become a governess to the children of a Naval officer widower.
Director: Robert Wise | Stars: Julie Andrews, Christopher Plummer, Eleanor Parker, Richard Haydn
Votes: 197,388 | Gross: $163.21M
Year LGC NFR year 1965 61 2001
27. The Bride of Frankenstein (1935)
G | 75 min | Drama, Horror, Sci-Fi
Mary Shelley reveals the main characters of her novel survived: Dr. Frankenstein, goaded by an even madder scientist, builds his monster a mate.
Director: James Whale | Stars: Boris Karloff, Elsa Lanchester, Colin Clive, Valerie Hobson
Votes: 41,897 | Gross: $4.36M
Year LGC NFR year 1935 58 1998
28. Singin' in the Rain (1952)
G | 103 min | Comedy, Musical, Romance
A silent film production company and cast make a difficult transition to sound.
Directors: Stanley Donen, Gene Kelly | Stars: Gene Kelly, Donald O'Connor, Debbie Reynolds, Jean Hagen
Votes: 211,519 | Gross: $8.82M
Year LGC NFR year 1952 57 1989
29. Apocalypse Now (1979)
R | 147 min | Drama, Mystery, War
A U.S. Army officer serving in Vietnam is tasked with assassinating a renegade Special Forces Colonel who sees himself as a god.
Director: Francis Ford Coppola | Stars: Martin Sheen, Marlon Brando, Robert Duvall, Frederic Forrest
Votes: 586,476 | Gross: $83.47M
Year LGC NFR year 1979 57 2000
30. The Texas Chain Saw Massacre (1974)
18+ | 83 min | Horror
Two siblings and three of their friends en route to visit their grandfather's grave in Texas end up falling victim to a family of cannibalistic psychopaths and must survive the terrors of Leatherface and his family.
Director: Tobe Hooper | Stars: Marilyn Burns, Edwin Neal, Allen Danziger, Paul A. Partain
Votes: 131,202 | Gross: $30.86M
Year LGC NFR year 1974 57
31. Rebel Without a Cause (1955)
14A | 111 min | Drama
A rebellious young man with a troubled past comes to a new town, finding friends and enemies.
Director: Nicholas Ray | Stars: James Dean, Natalie Wood, Sal Mineo, Jim Backus
Votes: 81,037
Year LGC NFR year 1955 57 1990
32. Star Wars: Episode V - The Empire Strikes Back (1980)
G | 124 min | Action, Adventure, Fantasy
After the Rebels are brutally overpowered by the Empire on the ice planet Hoth, Luke Skywalker begins Jedi training with Yoda, while his friends are pursued by Darth Vader and a bounty hunter named Boba Fett all over the galaxy.
Director: Irvin Kershner | Stars: Mark Hamill, Harrison Ford, Carrie Fisher, Billy Dee Williams
Votes: 1,125,582 | Gross: $290.48M
Year LGC NFR year 1980 56 2010
33. North by Northwest (1959)
18A | 136 min | Adventure, Mystery, Thriller
A New York City advertising executive goes on the run after being mistaken for a government agent by a group of foreign spies.
Director: Alfred Hitchcock | Stars: Cary Grant, Eva Marie Saint, James Mason, Jessie Royce Landis
Votes: 289,480 | Gross: $13.28M
Year LGC NFR year 1959 54 1995
34. Rear Window (1954)
PG | 112 min | Mystery, Thriller
A wheelchair-bound photographer spies on his neighbors from his apartment window and becomes convinced one of them has committed murder.
Director: Alfred Hitchcock | Stars: James Stewart, Grace Kelly, Wendell Corey, Thelma Ritter
Votes: 429,043 | Gross: $36.76M
Year LGC NFR year 1954 54 1997
35. Mary Poppins (1964)
G | 139 min | Comedy, Family, Fantasy
In turn of the century London, a magical nanny employs music and adventure to help two neglected children become closer to their father.
Director: Robert Stevenson | Stars: Julie Andrews, Dick Van Dyke, David Tomlinson, Glynis Johns
Votes: 153,376 | Gross: $102.27M
Year LGC NFR year 1964 54 2013
36. Pinocchio (1940)
F | 88 min | Animation, Comedy, Family
A living puppet, with the help of a cricket as his conscience, must prove himself worthy to become a real boy.
Directors: Norman Ferguson, T. Hee, Wilfred Jackson, Jack Kinney, Hamilton Luske, Bill Roberts, Ben Sharpsteen | Stars: Dickie Jones, Christian Rub, Mel Blanc, Don Brodie
Votes: 126,639 | Gross: $84.25M
Year LGC NFR year 1940 53 1994
37. Willy Wonka & the Chocolate Factory (1971)
PG | 100 min | Family, Fantasy, Musical
A poor but hopeful boy seeks one of the five coveted golden tickets that will send him on a tour of Willy Wonka's mysterious chocolate factory.
Director: Mel Stuart | Stars: Gene Wilder, Jack Albertson, Peter Ostrum, Roy Kinnear
Votes: 172,724 | Gross: $4.00M
Year LGC NFR year 1971 52
38. The Seven Year Itch (1955)
PG | 105 min | Comedy, Romance
When his family goes away for the summer, a hitherto faithful husband with an overactive imagination is tempted by a beautiful neighbor.
Director: Billy Wilder | Stars: Marilyn Monroe, Tom Ewell, Evelyn Keyes, Sonny Tufts
Votes: 33,048 | Gross: $12.00M
Year LGC NFR year 1955 51
39. Rosemary's Baby (1968)
18+ | 137 min | Drama, Horror
A young couple trying for a baby move into a fancy apartment surrounded by peculiar neighbors.
Director: Roman Polanski | Stars: Mia Farrow, John Cassavetes, Ruth Gordon, Sidney Blackmer
Votes: 185,779
Year LGC NFR year 1968 51
40. West Side Story (1961)
G | 153 min | Crime, Drama, Musical
Two youngsters from rival New York City gangs fall in love, but tensions between their respective friends build toward tragedy.
Directors: Jerome Robbins, Robert Wise | Stars: Natalie Wood, George Chakiris, Richard Beymer, Russ Tamblyn
Votes: 93,993 | Gross: $43.66M
Year LGC NFR year 1961 51 1997
41. Dirty Harry (1971)
18A | 102 min | Action, Crime, Thriller
When a madman calling himself "the Scorpio Killer" menaces the city, tough as nails San Francisco Police Inspector "Dirty" Harry Callahan is assigned to track down and ferret out the crazed psychopath.
Director: Don Siegel | Stars: Clint Eastwood, Andrew Robinson, Harry Guardino, Reni Santoni
Votes: 139,699 | Gross: $35.90M
Year LGC NFR year 1971 51 2012
Tell Your Friends