Recently, while reading Vanity Fair’s account of the wisdom of crowds and how that wisdom can be mined to predict the Oscars, I found myself nodding along with the piece. Yes, I am a firm believer that crowdsourcing can be a great approach to solve thorny problems. Like Vanity Fair, I am often skeptical of the “expert”. Relying on expert judgment, though sometimes necessary, often leads one to rely on ill-formed opinion—witness the forecasts of the Etruscan Haruspex if you’d like a particularly gut-wrenching image of pre-classical expert opinion. Of late, crowd-sourced wisdom is piling up front page stories of success. The famous Netflix Prize found a suggestion algorithm better than Netflix’s own solution, and the 2008 and 2012 election forecasting case studies again illustrate the power of crowdsourcing. We’ve at least sampled the Kool-Aid too: in our own Oscar Forecasting model, we use data obtained from crowdsourcing (notably Intrade futures) for a part of our forecast. Just having a crowd, however, is not a silver bullet for a great solution.
Essential to analyzing the success and value of crowd-sourcing is a firm grasp of one thing – the crowd. The question is – who comprises the crowd. In some sense, scientific research is a crowdsourcing exercise. Vast armies of physicists are busily toiling on many vexing problems central to the understanding of the universe. Any one of them could publish the next breakthrough paper. No one would suggest, however, that the general public should be asked to crowdsource high-energy physics or cosmology (or even basic physics). In that event we’d probably have textbooks telling us that electricity flows faster in wires that go downhill. It does not; and we don’t recommend you try testing this at home.
The Oscars themselves rely on crowdsourcing, limited to the universe of the members of their club. Yes, the Academy is a large crowd of Hollywood types, but this is just one level of the hierarchy of crowds. On one extreme of the hierarchy, the ultimate crowdsourced solution would be to pick the best picture of the year based on the box office: whatever film had the highest gross is the best film as voted for by the great unwashed masses. In 2012, we would be honoring The Avengers. In fact, no single nominee is in the top 10 for 2012 Box Office; Lincoln ranked highest at 14. Most people, even those unwashed masses perhaps, would see this as a ludicrous way of determining the best film of the year. We could down-select, within the hierarchy, to the crowd who is passionate about movies, have an internet connection, and the free time to comment on them semi-anonymously on websites. This could have be represented a few different ways, including Rotten Tomatoes scores – where Argo would win with a 96% rating. And, one level more critical would be the crowd of IMDB commenters – serious film-goers or industry types who take the time to both create IMDB accounts and comment on films. In this crowd we’ve had some self selection (similar to the self-selection of physicists who publish scholarly articles), and probably, in some sense, a keener eye for cinematographic fare. This would suggest that this large group of highly engaged people would be a crowd that would likely excel at divining the entelechy of truly great cinema.
Do I suggest that the aggregate of IMDB die-hards could be better at selecting which films are the best of the year than the vaunted Academy of Motion Pictures Arts and Sciences? Yes, indeed I do. Does this mean that this crowd (and this model) is the best source for Oscar predictions? Not so fast.
The Academy has a crowd of their own that, in all likelihood, has a different make up (in every sense: demographically, geographically, etc.) than the IMDB crowd. Not to mention that forecasting the Oscars does not require determining which is the best film or best actor; it requires determining who the Academy thinks is the best film or actor or whatever.
We see this phenomenon of different crowds coming to different conclusions in the awards season, even if we haven’t called it as such. The Golden Globes are a long standing Hollywood tradition (if only for the open bar). In these awards members of the Hollywood Foreign Press decide on the best in cinema. The journalists who vote on these awards are often not members of the Academy, so any time the Globes and Oscar agree, it is because the majority of members in each independent group have used at least similar criteria to judge what makes the best film, performance, etc. It seems likely that at one point, both these crowds shared a common criteria for greatness that has since changed as the two awards have diverged in the past decade or so. This is not necessarily surprising as the Academy membership has changed, and one might expect that actors, for example, have a different idea of what is great acting than a journalist. Perhaps it is surprising that they both once agreed so much. I’d say that the recent predictive power of the Critics Choice Awards could be as much of a stroke of luck as the Globe’s once was.
If the IMDB crowd and the Golden Globes (or Critics Choice) crowd were the only sources of determining a reasonable Oscar night forecast, things wouldn’t be so bad–the predictions based on IMDB were pretty accurate according to Vanity Fair. We do, however, get signals from the actual Academy crowd earlier in Awards season: the “guild” awards.
A little lesson on Oscar voting: The Academy has divisions and each division votes on the relevant awards: actors vote for Best Actor/Actress, directors for Best Director, etc. Everyone votes for Best Picture. Those same divisions are often members of the guilds who host these guild awards. Actors (though a larger crowd of them) vote for the Screen Actors Guild Awards and the directors vote for the Directors Guild of America Awards. Producers and writers have their own awards too. These awards are strong indication of how the members of the given divisions feel. If the actors voted for a particular supporting actress for a SAG award, I’d feel pretty confident that the smaller sample of actors in the Academy won’t feel completely different. For this reason, we also include these other awards in our forecast of the Oscars.
In forecasting the Oscars, as in many arenas of data science, two approaches are often complementary and can be used together. Crowd-sourcing, included in our model through Intrade data, and other awards/indicators make a more confident prediction. If we just had crowdsourcing data, we’d be missing what some members of the Academy might be hinting before the event. If we just use other awards/indicators we might miss a groundswell building behind a particular film/event a la Crash’s surprise win several years ago.
In our forecast we’ve tried to take all data sources that we think can give us the best prediction. We’ll know for sure (or as sure as statistics ever gets) in a few days.