Wednesday, April 13, 2011

Which analysts are the biggest sandbaggers - charted

Over the past year or so I've been tracking all of us analyst performance in forecasting Apple's financial metrics, and ranking us based on the 6 or 7 categories of estimates compiled by Fortune's Apple 2.0 blogger Philip Elmer-DeWitt, and the outcome has always been a consistent underperformance by pros (see hereherehere and here). The comparison and friendly "competition" has almost become laughable, if it weren't for the serious amount of capital that these "professional" analysts look over, and thus the effect of their cluelessness on Apple's share price gets felt.

However, all this time I've been applying a somewhat forgiving methodology on my rankings. By averaging out all the categories with equal weights, the resulting score improperly reflects the relative importance and sensitivity on the stock price of these variables. It should be clear to all investors that forecasting earnings and revenue is most critical, while the number of iPods sold has very little effect (for quite a few years now) on Apple's financial performance. Yet by applying the same weight to these, the effect of the most important metrics gets watered-down by the less important ones so the analyst scores and thus the rankings do not reflect what investors should be focusing on out of all the stuff analysts throw out there.

To correct this I've now decided to concentrate on those two most important metrics, EPS and revenue, when calculating the scores. This is what investors really want to see if Apple beats the expectations of, and by how much. Granted, gross margins and the iPhone units metrics are almost as important, but these already have such a big effect on the EPS and revenue forecasts that including them in the score would be like counting them twice towards the final outcome. If either of these are a blowout or a disappointment, it will certainly show in the revenue and EPS results, so they're already being accounted for. So there.

Also, for the cumulative rankings I've only used the latest 4 quarters worth of data compiled by PED, mostly because that's when the blogger analysts have come out in force with their estimates and truly made this a representative competition. Still, I feel like the work done by us three pioneering blogger-analysts, Bullish Cross' Andy Zaky, Financial Alchemist's Turley Muller, and myself, as well as the gracious opportunity given to us by PED over more than 2 years now, should not go to waste. Therefore I've decided to expand the scope of these scores and rankings to cover the whole period of smackdowns, going back to the early "analysing the analysts" posts by PED (and Andy just once before that).

One thing I'd like to make crystal clear is that this reassessing of the scores and rankings is in no way motivated to make my estimates, or those of us bloggers as a group, look better or rank higher, even if the actual result is as such. We certainly don't need that to come out on top. Rather, this is just me trying to get the most relevance out of the data for investors. Neither does it mean I'm discarding all the other variables. All that rich history of unit estimates comes handy when assessing the speculative musings of any of these analysts in the future (e.g. Gauna's recent shenanigans). Keeping tabs on for example their iPhone or iPad units accuracy should give pause to anyone thinking about taking those speculations too seriously. But I'll leave that to each investor, and as I said, I'll build my ranked list based on the EPS and revenue figures.

For this, I've taken the trouble of recompiling all the data going back three years, as published by PED (and one early article by Andy):

FY2008: Q2, Q3, Q4
FY2009: Q1, Q2, Q3, Q4
FY2010: Q1, Q2, Q3, Q4
FY2011: Q1

I've loaded all these estimates as well as the actual results as reported by Apple on these 6 or 7 categories over these 11 quarters and up to 51 different analysts into a nifty database model, so I can now conveniently query, slice and dice the data at will (Excel was already starting to show its limitations). More than 1500 data points in all. Let's take a look.

First, here's an overall picture of the accuracy of bloggers and pros by category, showing an interesting pattern:

Average Accuracy ( 1 - abs.error% ) by Category

Category  Bloggers      Pros  Err. reduction
--------  --------  --------  --------------
     EPS     94.1%     85.0%           60.9%
 Revenue     96.7%     93.8%           46.5%
 iPhones     89.9%     88.0%           16.1%
   iPods     95.1%     93.2%           28.2%
    Macs     95.3%     94.1%           19.1%
   iPads     86.3%     86.0%            2.1%
     GM%     96.8%     96.5%            8.2%
--------  --------  --------  --------------
All Cats     93.2%     92.0%           15.2%
 EPS+Rev     95.4%     89.4%           56.8%

Bloggers are better than pros in every category, but we're way, way better at EPS and revenue. Here's a bar chart of that (click to enlarge):

I'm aware from visual information rules and guidelines that the green continuous line (the error reduction) shouldn't have been a line but bars, as there's no continuity between categories. Yet, I liked how it looks as a line, I need it to stand in contrast with the bars behind so I'm sticking to it. *sosumi*

Next is a similar idea but this time I'm breaking it down by quarters, showing the whole history of available estimates and their changes over time:

Accuracy Over Time (all categories)

Period  Bloggers      Pros  Err. reduction
------  --------  --------  --------------
F08 Q2     95.8%     91.3%           52.2%
F08 Q3
F08 Q4     94.9%     90.5%           46.2%
F09 Q1     85.1%     87.9%          -23.4%
F09 Q2     91.8%     89.4%           21.9%
F09 Q3     97.3%     93.0%           60.7%
F09 Q4     95.1%     91.3%           44.1%
F10 Q1     97.5%     93.0%           63.5%
F10 Q2     92.4%     88.5%           33.5%
F10 Q3     95.7%     93.9%           28.3%
F10 Q4     90.1%     90.1%            0.6%
F11 Q1     96.1%     91.0%           56.4%

Notice only one period in which we did worse than pros, and one which was a tie. All the others we were significantly better. Here's a chart (click to enlarge):

And now I'll focus on the critical EPS and revenue metrics. Here's that over time:

Accuracy Over Time (EPS+Revenue)

Period  Bloggers      Pros  Err. reduction
------  --------  --------  --------------
F08Q2      95.4%     93.3%           31.2%
F08Q4      96.4%     93.8%           41.3%
F09Q1      96.3%     87.4%           71.0%
F09Q2      96.3%     87.5%           70.0%
F09Q3      98.5%     93.5%           77.4%
F09Q4      89.9%     86.5%           25.6%
F10Q1      96.8%     92.9%           54.2%
F10Q2      88.0%     82.3%           31.9%
F10Q3      96.2%     92.2%           50.9%
F10Q4      97.2%     91.2%           68.2%
F11Q1      96.9%     87.2%           75.7%

The chart:

Cool! Feels good to get within 5% and even 3% with such consistency, but even better to cut the pros error in half and sometimes even down to less than a fourth. But doing this on the critically relevant EPS and revenue figures is what smart investors care for.

The only thing about these figures is that because they're expressed as percentages in order to be able to compare and combine the different metrics each with different units (EPS in dollars, revenue in $billions, units in millions of units, GM is a ratio), then the actual size of the errors by pro analysts kind of gets lost. Another distinction is that since these are all intended as measures of accuracy, it's not shown in which direction is the miss, it's just the average percentage amount of a miss, be that positive or negative, they all add up. But it's quite revealing to also consider the net size and direction of the average miss, and to do that I'll now keep the sign of each error and thus the positives will cancel some of the negatives, and what's left is a measure of how consistently skewed (or not) in whichever direction (always down!) are the estimates, which I call bias (you can see a more detailed accuracy/bias analysis on my own estimates going back more than 6 years here). Also, I'd like to start looking at individual performances now.

So here's the actual net average dollar amount of EPS by quarter, broken and sorted by analyst, that they tend to miss including the direction (all of us aim low but pros are extremists). Notice I include the number of estimates involved in each analyst's score to show that these are not one or two-times flukes, but persistent sandbagging. The last column really puts this in context, showing the net amount algebraically added over all the estimates made by each analyst. I've aggregated those analysts that have not made enough estimates for a consistent bias evaluation as "other" (includes most of the new bloggers like Asymco's Horace Dediu and the AFB gang who as a group show a remarkably neutral bias). So here we go:

Bullish/Bearish EPS Bias (listed analysts with at least 4 EPS estimates)

Rank  Analyst      AvgBias($)  No.ofEsts  AccumBias($)
----  -----------  ----------  ---------  ------------
   -  Other blogs       -0.04         15         -0.53
   1  Zaky              -0.05         11         -0.54
   2  Tello             -0.09         11         -0.94
   3  Muller            -0.09         10         -0.94
   4  Leitao            -0.11          4         -0.43
   5  Cabot             -0.15          4         -0.59
   6  Abramsky          -0.31         11         -3.37
   7  Marshall          -0.33          7         -2.29
   8  Reiner            -0.35          9         -3.19
   9  Craig             -0.37          9         -3.33
  10  Sacconaghi        -0.40         10         -3.99
  11  Rakesh            -0.41          5         -2.04
  12  Wu                -0.41          9         -3.72
  13  Gardner           -0.42          8         -3.39
  14  Chokshi           -0.43          4         -1.71
  15  Reitzes           -0.48         11         -5.26
  16  Munster           -0.50         11         -5.49
  17  Huberty           -0.50         10         -5.01
  18  Shope             -0.51          5         -2.56
  19  Reid              -0.53          7         -3.68
  20  Fidacaro          -0.53          7         -3.68
  21  Fearnley          -0.55          4         -2.19
  22  Moskowitz         -0.56          5         -2.81
  23  Um                -0.61          4         -2.45
  24  Whitmore          -0.63          4         -2.51
  25  Bachman           -0.64          6         -3.85
  26  McCourt           -0.81          6         -4.83
   -  Other pros        -0.65         31        -20.26
      --------------  -------  ---------  ------------
      All blogs         -0.07         55         -3.97
      All pros          -0.50        183        -91.61
      --------------  -------
      Bias reduction    85.6%

It's amazing to me that such a highly regarded analyst as Gene Munster, and considered by many the most bullish on Apple, has managed to underestimate EPS by a whole 5 bucks and a half over less than 3 years. That's a whole $100 worth on AAPL if you price it at an 18x multiple. And that's our bullish guy. Mr. McCourt on the other hand, managed to misplace almost 5 bucks in only a year and a half. Give it another year and he'll easily be 8 bucks behind, or near $150 on the stock by a conservative multiple. But he'll go on giving interviews on CNBC and such.

The chart is a bit dizzying:

Same thing with revenue sandbagging, no end to their pessimism:

Bullish/Bearish Revenue Bias (listed analysts with at least 4 revenue estimates)

Rank     Analyst   AvgBias($b)  No.ofEsts  AccumBias($b)
----  -----------  -----------  ---------  -------------
   -  Other blogs        -0.19         15          -2.83
   1  Leitao             +0.22          4           0.87
   2  Tello              -0.12         11          -1.35
   3  Muller             -0.16         10          -1.63
   4  Cabot              -0.18          4          -0.73
   5  Zaky               -0.20         11          -2.22
   6  Abramsky           -0.44         11          -4.87
   7  Reiner             -0.49          9          -4.45
   8  Munster            -0.81         11          -8.95
   9  Craig              -0.83          9          -7.50
  10  Wu                 -0.86          9          -7.78
  11  Reitzes            -0.87         11          -9.62
  12  Sacconaghi         -0.89         10          -8.93
  13  Gardner            -0.90          8          -7.23
  14  Marshall           -0.97          7          -6.78
  15  Shope              -0.98          4          -3.92
  16  Huberty            -1.02         10         -10.23
  17  Fidacaro           -1.06          7          -7.39
  18  Moskowitz          -1.10          5          -5.50
  19  Chokshi            -1.21          4          -4.82
  20  Bachman            -1.21          6          -7.27
  21  Rakesh             -1.21          5          -6.06
  22  Fearnley           -1.30          4          -5.19
  23  Whitmore           -1.36          4          -5.42
  24  Reid               -1.38          5          -6.89
  25  Um                 -1.42          4          -5.66
  26  McCourt            -1.65          6          -9.87
   -  Other pros         -1.32         31         -40.90
      --------------  --------  ---------  -------------
      All blogs          -0.14         55          -7.89
      All pros           -1.03        180        -185.23
      --------------  --------
      Bias reduction     86.1%          

One billion dollars short, each and every quarter, on average for the last 2-3 years. It may not seem much right now compared to these $25b quarters, but in 2008 quarterly revenue was only around $7.5-8b. And because recently sales have accelerated yet the analyst's percentage error remains constant or keeps increasing, that means in absolute dollars amount it's getting worse every year and every quarter. Once again Mr. McCourt is exemplary with his $1.65b average bias every quarter for the last year and a half. I suspect the pro average quarterly revenue miss for this year and the next will be nearing well over $2b.

PED (Jan 2011 post): Bloggers vs. pros: The $2 billion gap
PED (Apr 2011 post): Bloggers v. Pros: A $2 billion gap

On the other hand, well deserved kudos to Robert Paul Leitao for his courage in taking up the sole but oh-so-very-slightly bullish position over the last 4 quarters. And here's the chart:

Makes one wonder how far off them Wall Street consensus estimates for 18% revenue growth and 15% EPS growth in 2012 will turn out. If they haven't been able to get within 10% of the most recent quarter, how will they fare on stuff that's more than a year away? For now, they're 25% short of my 2012 EPS estimate of $35.76. If I'm roughly right and their past-quarter performance is to remain around 10% short, then over the next few quarters they'll need to raise the 2012 estimates by about 20%, putting further pressure on the declining forward P/E if the current stock price was maintained.

Finally, the whole list of all 51 analysts, including some obscure one-timers, ranked by the combination of EPS and revenue accuracy, which is the one I'll keep tracking regularly (although I'll keep an eye on the other things too):

EPS and Revenue Combined Accuracy ( 1 - abs.error% )

Rank  Analyst     Accuracy  No.ofEsts.
----  ----------  --------  ---------
   1  Muller         96.3%         20
   2  Fosberg        96.2%          6
   3  Hildebrand     96.2%          6
   4  Tello          95.7%         22
   5  Mihalache      95.4%          6
   6  Dediu          95.2%          6
   7  Zaky           95.1%         22
   8  Cabot          94.9%          8
   9  Leitao         94.7%          8
  10  Smellie        93.1%          4
  11  Abramsky       92.5%         22
  12  Chokshi        92.3%          8
  13  Klugman        92.2%          1
  14  Beauch         92.0%          2
  15  Cihra          92.0%          6
  16  Kumar          91.8%          6
  17  Reiner         91.5%         18
  18  Gillis         91.4%          2
  19  Rakesh         91.3%         10
  20  Gauna          91.2%          2
  21  Craig          90.8%         18
  22  Schackart      90.5%          4
  23  Sacconaghi     90.4%         20
  24  Abramowitz     90.4%          2
  25  Marshall       90.4%         14
  26  Ghai           90.2%          6
  27  Fearnley       90.0%          8
  28  Wu             90.0%         18
  29  Moskowitz      89.8%         10
  30  Hoffman        89.5%          6
  31  Gardner        89.5%         16
  32  Fidacaro       89.4%         14
  33  Whitmore       89.3%          8
  34  Haley          88.8%          2
  35  Walkley        88.5%          4
  36  Susanto        88.4%          2
  37  Shope          88.2%          9
  38  Ernst          88.2%          4
  39  Huberty        88.2%         20
  40  Um             88.0%          8
  41  Munster        88.0%         22
  42  Reid           87.9%         12
  43  Reitzes        87.8%         22
  44  Bachman        87.6%         12
  45  White          86.8%          2
  46  Sutherland     86.7%          2
  47  McCourt        85.5%         12
  48  Power          84.6%          2
  49  Misek          84.3%          4
  50  Wolf           81.7%          4
  51  Bailey         80.2%          1
      ------------  ------  ---------
      All bloggers   95.4%        110
      All pros       89.4%        363
      ------------  ------
      Err. reductn   56.8%

A well deserved congrats to Turley, Jeff, and Dennis (told ya!), and to all bloggers for consistently getting it right where it matters most: the revenue and EPS forecasts. And a huge thank you to Philip Elmer-DeWitt and Fortune (as well as their readers) for making all this possible. Keep it up!


David Emery said...

Of course, this begs the question why Wall Street firms aren't beating down your doors. Thanks for providing this, and keep up the good work :-)

Andy M. Zaky said...

@ David - I don't know. I ask that f'ing question every day. I'll take Huberty's job! Morgan Stanley can do so much better!

To Daniel - DAMN good article. WOW! What an effort. Thanks for this! Really. Thank you!

JavaJack said...

Great article. You put a lot of effort into this .

Jon T said...

Well done for illuminating what we have all felt for the last few years..

Now, the next step is for Apple to recognise this and invite you three leaders to ask questions at the earnings calls instead of the daft professionals.

Michel C said...

Great work

Yep we should nominate you as the representative of the afb. Poll a question and have you ask it to tc.

Suddy said...


Just curious what is your take on the AAPL stock price in the next 2 years. I am long AAPL stock that I got in the ~ 200's, so I am good there and I am long term bullish on AAPL - of course!

The reason for my question is more with the 2013 way out of the money leaps that I own.

What are your thoughts. As much as I hate/ care less about technical analysis or even thinking about technicals, I just can't understand what is holding back AAPL stock for the last 3+ months?

Daniel Tello said...

Suddy, I'm not too fond of TA either (and long-term is irrelevant) but I see AAPL trading over $600 by Jan 2013, and during 2012 can't see it ever dipping under $450. Last 3 months is just AAPL trading silly, which sometimes happens in the short term. This silliness only affects those of us holding May-Jul 2011 options. Fortunately you're long shares and LEAPS13 (any strike currently available will gain nicely). If you're too scared you can always throw OTM puts for some protection. Or try on a collar.

Of course this is just my opinion and highly subjective guess, not guaranteed nor intended as financial advise. Good luck!

Dennis Hildebrand said...

Excellent work Daniel.. nice consolidation of all the info.. Your hard work is most appreciated. Yeah - you told me alright!


KGB said...

...and the ultimate question remains....why are these professional analysts moving in a herd and consistently below the amateurs? It all looks a bit staged. Perhaps the pros take the mindset that being (ultra)conservative and low-balling revenue/eps etc, etc every quarter for AAPL, they are being robustly cautious reflecting a mature and trustworthy mirror to their investors!
Thus, why the consistent discrepancy? We assume the pros have access to the same figures/data/parameters that the amateurs work with. Or is this being too simplistic on my part?

John P. said...

What a massive project. Thanks form all of us novices... JWP