February 17, 2005

How to Lie with Statistics, I

One of the earliest topics in our sabermetrics course is “how to lie with statistics,” or, more appropriately, how to know when others are using and abusing statistics in attempt to lie to you.  To find examples of such, you need only to open the pages of your favorite sports daily.  Bill James noted this almost 25 years ago:  “Sportswriting draws on the available evidence, and forces conclusions by selecting and arranging that evidence so that it points in the direction desired.”  Not much has changed in the quarter-century, as evidenced by today’s Chicago Tribune.

Writing about White Sox pitchers Mark Buehrle and Freddy Garcia, Bob Foltman of the Trib notes that skipper Ozzie Guillen will “pace his top two starters and have them ready for postseason action in October.”  Taking a pass at the meatball that is Guillen’s incredible optimism regarding the playoff chances on the South Side, Foltman instead takes a hack by proposing that an innings reduction will be beneficial for Buehrle.  After all, Mark led the American League in innings pitched last year, so surely he’ll show ill effects in 2005, right?  The reader is to be swayed by Foltman’s data on Buerhle’s skills – particularly that the southpaw gave up career highs in home runs and hits allowed last season.

The short piece holds two great lessons for those aspiring to present misleading statistics:  cherry-picking evidence and ignoring appropriate control (comparison) groups.

Selective Evidence:

The first warning flag should pop up when the reader sees no mention of Cha-Cha Garcia after the article’s first sentence.  Did Freddy not fit Foltman’s thesis?   Probably.  Is it misleading to ignore him for the rest of his article?  More than probably.  But there’s such good bastardization of Buehrle’s data that we can ignore Garcia here as well.

We know the most important, trackable statistics for pitchers include walks, homeruns, and strikeouts.  Foltman only gives us two of the three.  No mention of Bulldog’s career-high strikeout rate in 2004.
 
Perhaps Foltman is not comfortable presenting such ‘rate statistics’ as K/IP or K/BF.  After all, he chooses to cite Buehrle’s homerun and hits allowed – the major crux of his argument – as ‘count’ statistics:  33 homers allowed and 257 hits allowed in 2004, both career highs.  But these numbers are meaningless, as Buerhle also posted a career high in innings last season (which of course is the entire thesis of his article!).  If Foltman thinks that hits are important, then by looking instead at batting average against (basically hits allowed ‘adjusted for’ batters faced), he could see that Buerhle actually fared the same in 2004 as he did in 2003 (.253 vs. .256, respectively).  Rate stats are clearly the more appropriate metrics here, and Foltman misleads by not using them.

Control Groups:

Looking at Buehrle’s homerun rate, we see that perhaps Foltman is on to something:

2001: 0.11 HR/IP
2002:  0.10
2003:  0.10
2004:  0.13

Aha!  A 30% increase in homeruns allowed.  Perhaps the workload IS catching up to the southpaw.  But, as anyone who watched more than a dozen games at US Cellular last season knows, it is misleading to compare White Sox homerun data from 2004 to pre-park-renovation levels.  It’s apples to churros – the new New Comiskey was the easiest park to homer in last season, and this cannot be ignored.  One appropriate control or comparison group for Mark’s 2004 data would be his homerun rate in games outside of Chicago:

2002:  0.137 HR/IP
2003:  0.078
2004:  0.095

The “noise” about the data is larger, because the sample size of innings pitched is smaller, but from these three datapoints there is no evidence that Buehrle has been Radkesque when it comes to gopherballs.  Another appropriate comparison group would Buehrle’s buddies in the stable of starting pitchers – how were their homerun numbers affected by the renovation of the upper deck at the Cell?  I’ll leave that task to the reader.

These aren’t the only things to criticize about the article.  Foltman creates the ultimate boner by citing a ‘fact’ without even researching its veracity.  He suggests that Buehrle gave up more dongers because he gave up more flyballs.  But a quick look at espn.com will tell you that Mark’s groundball / flyball ratio has been consistently around 1.5 for the past three seasons.

Look, Foltman is to be commended for taking an observation and forming an interesting hypothesis:  does workload in one season affect a pitcher’s performance in the next?  It’s a question that could be answered with sabermetric methods – define and quantify ‘workload,’ chose the appropriate outcome which would show an effect, collect an unbiased sample of data, and conduct and interpret the analysis.  It’s an example of sabermetrics, and you could do it well.  Unfortunately it’s also an example of sportswriting, and Bob Foltman does that well too.

Home