Nate Silver got his professional forecasting start in baseball. He devised a successful performance forecasting system based on complex statistical algorithms, but he also says that some of the forecasting systems that incorporated the subjective judgments of scouts were more accurate. Silver devotes a section of his book, The Signal and The Noise, to a discussion of baseball scouting versus number crunching. Here's Silver commenting on statistician biases:
[S]tatheads can have their biases too. One of the most pernicious ones is to assume that if something cannot easily be quantified, it does not matter. In baseball, for instance, defense has long been much harder to measure than batting or pitching. In the mid-1990s, Beane’s Oakland A’s teams placed little emphasis on defense, and their outfield was manned by slow and bulky players, like Matt Stairs, who came out of the womb as designated hitters. As analysis of defense advanced, it became apparent that the A’s defective defense was costing them as many as eight to ten wins per season, 33 effectively taking them out of contention no matter how good their batting statistics were. Beane got the memo, and his more recent and successful teams have had relatively good defenses. These blind spots can extract an even larger price when it comes to forecasting the performance of minor-league players. [...]
The key to making a good forecast, as we observed in chapter 2, is not in limiting yourself to quantitative information.
This reminded me of the problem with trying to operationalize and measure everything done in psychotherapy. If it can't be measured, we shouldn't do it, or so the researchers might have us believe. This despite a great deal of research showing that outcomes with psychodynamic therapies are as effective as so-called symptom-focused, evidence-based techniques.