The Netflix Prize

What happened

In October 2006 Netflix released over 100 million ratings, with their dates, from over 480 thousand anonymous subscribers on nearly 18 thousand titles, collected between October 1998 and December 2005. It offered a one million dollar Grand Prize to the team that improved the accuracy of its Cinematch recommender by a further 10 percent. A team called BellKor's Pragmatic Chaos won in 2009.

The psychological idea

The contest was an open prediction competition. Accuracy was scored by the root mean squared error between predicted and actual star ratings on over 3 million withheld recent ratings. Netflix reported Cinematch at 0.9514 on the quiz subset, 9.6 percent better than predicting each movie's average. The framing treated recommending as the problem of predicting how many stars a person would give a film.

In practice

Netflix authors say the company relied heavily on star-rating prediction when its main business was DVDs by mail, and that algorithms from the competition are still used in production to predict ratings. A 2012 news report quotes Netflix as saying it tested the newer methods offline and the gains did not seem to justify the engineering effort, and that its focus had moved on. These are company statements, not independent findings.

Why it mattered

The prize was a public release of a large body of customer ratings, together with a recommendation problem, to outside researchers. Netflix's own paper reported that by June 2007 over 20 thousand teams had registered. Netflix's later account also describes a change in the task: from predicting star ratings toward using what members actually watch.

Evidence behind the claims

• In October 2006 Netflix released over 100 million ratings with dates from over 480 thousand anonymous subscribers on nearly 18 thousand titles, collected October 1998 to December 2005, and withheld over 3 million recent ratings as a qualifying set.

  DocumentedChecked at full-text levelWhere: Section 2, The Netflix Prize[1]

• Netflix reported Cinematch at RMSE 0.9514 on the quiz subset, 9.6 percent better than predicting movie averages, and set the Grand Prize for a further 10 percent improvement.

  DocumentedChecked at full-text levelWhere: Section 2[1]

• MIT Technology Review reported in October 2006 a one million dollar prize for increasing the accuracy of the Netflix recommendation system by 10 percent by 2011, with an annual 50,000 dollar progress prize.

  DocumentedChecked at page levelWhere: Article text as returned by the fetch tool[2]

• The 2009 grand prize went to a team called BellKor's Pragmatic Chaos.

  DocumentedChecked at page levelWhere: Article text[3]

• BGR quotes Netflix as saying it evaluated the new methods offline and the additional accuracy gains did not seem to justify the engineering effort, and that its focus on personalization had shifted by then.

  Company self-reportChecked at page levelWhere: Article text quoting Netflix's 2012 blog post, which was not opened[3]

• Netflix authors say recommending was historically treated as predicting star ratings, that algorithms from the competition are still used in production to predict ratings, and that viewing data now give much better ways to help people find videos than focusing on a high predicted star rating.

  Company self-reportChecked at full-text levelWhere: Section 2, The Netflix Recommender System[4]

• By June 2007 over 20 thousand teams had registered from over 152 countries.

  DocumentedChecked at full-text levelWhere: Section 5, Prize status, June 2007[1]

People involved

• James Bennett. Netflix, co-author of the 2007 description of the Prize
• Stan Lanning. Netflix, co-author of the 2007 description of the Prize

Course connections

• OpenStax Psychology 2e, section 2.3: Analyzing FindingsContext. The section discusses this topic.

  Opened the page: it says correlations have predictive value, with a grades example, which is the loose link to predicting ratings from past data.

• OpenStax Psychology 2e, section 6.3: Operant ConditioningContext. Nearest related section; it does not discuss this topic by name.

  Nearest related section: operant conditioning. The Prize concerns prediction accuracy, not conditioning.

Course relevance is not evidence for the historical or corporate claims.

Sources and inspection

Inspection statements below preserve the original reading scope. A source marked as not verified supports no claim by itself.

1. [1]
   Bennett, J., & Lanning, S. (2007). The Netflix prize. Proceedings of KDD Cup and Workshop 2007 (ACM). https://www.cs.uic.edu/~liub/KDD-cup-2007/proceedings/The-Netflix-Prize-Bennett.pdfBennett, J; Lanning, S. Proceedings of KDD Cup and Workshop 2007 (ACM). 2007. Journal article. Company document.

   Company document read. Read all four pages of the Netflix-authored paper (abstract, section 2 on the Prize, section 3, section 5 on status in June 2007) from the PDF.

2. [4]
   Gomez-Uribe, C., & Hunt, N. (2015). The Netflix recommender system: Algorithms, business value, and innovation. ACM Transactions on Management Information Systems, 6(4), Article 13. https://gwern.net/doc/reinforcement-learning/exploration/2015-gomezuribe.pdfGomez-Uribe, C; Hunt, N. ACM Transactions on Management Information Systems. 2015. Journal article. Company document.

   Company document read. Read pages 1 to 10 of the 19-page PDF copy (abstract, sections 1 to 4.2). Written by Netflix employees; the copy is hosted by gwern.net and carries the ACM publication line, December 2015.

3. [3]
   Graziano, D. (2012, April 13). Netflix paid $1 million for a recommendation algorithm it never used. BGR. https://www.bgr.com/2012/04/13/netflix-paid-1-million-for-a-recommendation-algorithm-it-never-used. BGR. 2012. News. Secondary.

   Reference article read. Read the news report as returned by the fetch tool. It quotes Netflix's April 2012 blog post; the reviewer could not open the Netflix post itself (HTTP 403 for netflixtechblog.com), so the quoted wording rests on BGR alone.

4. [2]
   Greene, K. (2006, October 6). The $1 million Netflix challenge. MIT Technology Review. https://www.technologyreview.com/2006/10/06/273459/the-1-million-netflix-challenge. MIT Technology Review. 2006. News. Secondary.

   Reference article read. Read the 2006 news article as returned by the fetch tool: prize terms, 2011 deadline, progress prize and the 103 million and 100 million rating figures.

Access note: Netflix's 2012 blog post on the unused winning algorithm returned HTTP 403 for both the researcher and the reviewer; its wording is taken from one news report (BGR). Original press release not opened.

Related cases and psychology

• Netflix describes its recommender system and its business value
• Deep Neural Networks for YouTube Recommendations