Skip to content

Comparing the Projection Systems- WTF, PECOTA?

October 10, 2009

Vegas Watch has crunched the numbers and give us the results for the 2009 projection systems.  The great projection race gives us a pretty good example as to why we cannot trust a single projection system for predicting future performance.  PECOTA, which is perhaps the most famous of all the projection systems out there, comes in dead last this year after having a remarkable run from 2005-2008.  Marcels comes in third with an RMSE of 9.77 and is the most simple projection system you could possibly imagine- so keep in mind that no matter how detailed the projection system is, chances are it won’t be far off from the most simple in terms of overall results.

Speaking of the results for 2009, here they are:

pecotafail

CAIRO comes in first.  I wonder how MGL’s forecasting would have held up this year- it’s done a damn good job in years past.  There are three systems that have a tendency to get more exposure than others, so I thought I’d take a look at their three-year average:

Picture 2

CHONE looks to be the best of the three, but take that for what it’s worth- as PECOTA has shown us, projection systems won’t always be consistent in terms of its accuracy.  Personally, I’d recommend combining multiple projections (say, for example, CHONE, PECOTA and Marcels).  Even then you’ll probably only see a marginal improvement at best.

Advertisements
No comments yet

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: