Sports Law Blog
All things legal relating
to the sports world...
Wednesday, July 05, 2006
 
Good Player Personnel: Is it Moneyball, Scouting, or Sheer Luck?

Michael Bond of New Scientist Magazine has a fascinating interview with statistician Nassim Nicholas Taleb ("Life is Unpredictable: Get Used to It," 7/5/2006, subscription only), in which Taleb argues that humans are terrible at making predictions and that they tend to place false confidence in statistical models that yield results perhaps no better than mere guessing or sheer serendipity. Along those lines, the idea goes, we strive for explanations and patterns because they give us a greater sense of control, even when those explanations and patterns may be completely illusory. Here are some excerpts from his comments:
Our brains operate on autopilot most of the time . . . Our track record [for predictions] is quite dire. Look at the net, computers, lasers. The internet was designed as a military system, not for chat rooms. The person who first marketed computers didn't think he would sell more than five. The laser was designed by a physicist who had no idea how it might be used. You can't even forecast something that would affect us tomorrow - revolutions, wars, epidemics, political changes, economic variables . . . A forecast is irrelevant unless you have an error rate on it. But if this happened, these people would realise there was no point in forecasting, because their error rate would be so monstrous . . . [Successful forecasting is just luck], and you need a lot of luck to forecast things accurately.

[Question: But don't people make predictions based on history all the time?]

As well as our ability to concoct empirically flawed narratives to explain past events, there are biases in history that we don't seem to be aware of and that make us overestimate the causal links between events: for example, when you see only the winners and not the losers. When you look at the fossil record, you see only the species that left a fossil. You cannot make a generalisation of all species just from fossils - you have to take into account the species that left none. History has a lot of hidden pockets. You can't take it any more seriously than a visit to a museum.
What do Taleb's conclusions suggest about things like Moneyball and the rise of more complex-statistical models for player personnel decisions? Do Moneyball teams succeed for reasons that have nothing to do with empiricial approaches or taking advantage of "market inefficiencies"? Are "market approaches" then receciving too much credit (or blame) in management decisions (e.g., whether it is a good idea to draft or sign a particular player)? Of course, Nassim's conclusions don't validate traditional scouting models either, because they too would invite the same after-the-fact explanations for lucky results.

So statistics or not, is team management really about luck and coming up with an explanation later?





8 Comments:

"What do Taleb's conclusions suggest about things like Moneyball and the rise of more complex-statistical models for player personnel decisions? "

Well, so far his conclusions suggest nothing. He posits that results are unpredictable. Warren Buffet, Jerry West, Pat Riley and Brian Coleangelo would argue otherwise, with possibly more credibility on their side of the argument than Mr. Taleb.

Anonymous Paul Stevens -- 7/05/2006 9:23 PM  


Good points Paul, and thank you for commenting. I also find it hard to ignore repeated successes of certain people, and I can't believe that their success is merely the result of unpredictable luck. Having said that, and in fairness to Taleb, he seems to assert that unpredictability is more apparent on the macro-level of organizations/governing bodies than in individuals (but even then, there are some persuasive counter-examples).

Blogger Michael McCann -- 7/05/2006 9:54 PM  


What about Jens Lehmman, Germany's goal-keeper, who has so profitiently saved penalties on the world cup stage,FA cup, Premiership and at Champions League level? Luck, Taleb would say.

Well, Jens disagrees. One of his former coaches has a database with over 15.000 penalty takers and their percentages, favorite positions, etc., etc, and Jens gives him a call every time he plays a major game. To me, this seems the ultimate example of how forecast based on percentages may indeed overcome the limitations of luck.

Anonymous Luis Cassiano Neves -- 7/06/2006 5:42 AM  


luis cassiano neves,

You are right. Jens actually had a little note under his right sock and he pulled out the note after each penalty kick. (see--http://msn.foxsports.com/soccer/story/5745846).

Not having read the entire writing of Taleb, I hasitate to argue against it. I think a balanced use of stastistical data and good/educated business sense (luck could always help) is always better than sole reliance on the stats. I think we can say this not just about stats, but also about many of the business theories too. And we all know the familar saying goes....the numbers don't lie, right?

Anonymous Anonymous -- 7/06/2006 8:17 AM  


I think when you do anything you cant just look at it from one size. Using only statistics wont work by itself becuase you have to see the competition, and variables. Scouting by itself doesn't work because some times you are "looking to close". However, if you use both intelligently, you will be able to limit the luck factor a little bit more. In otherwords, there will always be luck but that doesn't mean you should not do anything and pick something out of a hat.

Having said that, I think a manager or coach only gets treated as good as his "luck is at that time". He can tell his players the right plays to run, what pitches to throw, but if they dont preform, he gets booted. If they do perform he is the best manager.
The world isnt fair.

Anonymous Anonymous -- 7/06/2006 9:51 AM  


When drafting players, the success rate varies from general manager to general manager. It also depends what you consider a "success." If a guy was taken purely to fill a need or role and he indeed did fill it than that is a success in a way. If your only talking about top picks that become all-stars than you have way more variables to consider. When a player gets that kind of money, how will it affect him? Is the team he is playing for going to give him the best opportunity to become a good player, like proper reps/minutes etc. I don't think it is a crapshoot, I just think when you have 30 opinions, not everyone is going to be right. Some are just right more than others.

Anonymous Ron Jumper -- 7/06/2006 12:25 PM  


I think the mainstream media has misread Moneyball as being a philosophy centered solely on statistics. Moneyball is a book, not the philosophy that the A's use to evaluate players.

I do not think there is a single person in a baseball front office (including Bill James) who thinks statistics are the only source of player evaluation, whether prospective or retrospective.

Essentially, Billy Beane was at the leading edge of a statistical wave that every single team in baseball subscribes to, albeit in different degrees. Lewis' book was about this approach, and because that was the theme of the book, many have the impression that the A's rely inordinately on statistical analysis.

The book wasn't focused on the "complete" approach to player evaluation . It was focused on one of the approaches (and a somewhat novel one at the time) that the A's use to evaluate players...and in particular, how they use that to build a winning ballclub with a modest payroll.

Scouts and traditionalists freaked out. Why? Because they see it as a job threat.

In reality, there is absolutely nothing in Moneyball that represents a new statistical analysis. People interested in baseball statistics (me included) have been working with those same concepts since the late 1980s...it's just that we were weirdo geeks from the perspective of the rest of the world.

The novelty of the book is that someone in a front office actually admitted to letting some of these statistical analyses creep into their decision-making process in a significant way. The sabermetricians finally broke through. :)

The people evaluating players (whether as a hobby or a profession) are not stupid enough to rely solely on numbers. Numbers are information, and some numbers are more reliable than others. Some scouts are more reliable than others too. Put them together in the right way and you get a better picture.

Branch Rickey had a staff statistician in his days with the Dodgers and Pirates. That was 50 years ago, but no one wrote a book about it. It is worth mentioning that Rickey also invented the modern farm system in baseball and probably employed more scouts than any other general manager of his time.

Blogger ChapelHeel -- 7/06/2006 1:23 PM  


It is interesting to see anyone call into question the Moneyball "system"--especially because the actual Moneyball class of players has failed so clearly. Of the seven first round picks, one is a budding star (Swisher, the no-brainer pick that few really questioned) and one is an effective major leaguer (Joe Blanton, who has a low strikeout rate that reduces his effectiveness). Of the others, none has made an impact in the majors. I could be remembering this incorrectly, but I believe that two more are marginal major leaguers on bad teams and the remaining three are not in the majors or out of the As' system.

In short, the results (about 1/3 accuracy) of the famed Moneyball draft are no different than if the A's had followed a more traditional approach, at least based on the success of the players that the As avoided. Luck, it seems, really does explain the outcome far better than any statistical analysis.

One other interesting quantitative result of the As' Moneyball system is that it appears to be very effective at identifying "Quadruple A" players--guys who are above-average minor league players but who can't make a consistent impact in the major leagues. I have no idea what this means for the viability of the Moneyball predictive model, but it seems to be good at predicting something, even if major league success isn't that thing.

In the end, the only thing Moneyball proves (and Taleb argues) is that accurate prediction requires more knowledge that most people can possess or than is available. The rest is intuition and guesswork. Warren Buffet may be demonstrably better than most people at choosing investments, but it's impossible to explain why.

Anonymous Anonymous -- 7/06/2006 7:31 PM  


Post a Comment