UC Berkley psychologist Philip Tetlock has been studying experts for 30 years. In what was perhaps his most widely publicized study, Tetlock found that political pundits fared no better than chance when predicting future events. The data also showed that better known pundits performed worse than lesser known pundits.
Tetlock says that the most popular pundits are ideologically-driven. That makes them more entertaining as media personalities, but adherence to The Grand Idea interferes with the capacity to modify opinions based on new information. Ideologies may leave their adherents feeling more certain about the accuracy of their own predictions, but ideologies actually diminish the accuracy of those predictions. The problem is similar to the processing difficulty identified by Drew Westen in his research with political partisans. New information that contradicts existing belief isn't processed rationally.
So, if so-called experts get it wrong so often, is anyone worth listening to? Tetlock says yes.
Q & A with Philip Tetlock:
What makes some forecasters better than others?
The most important factor was not how much education or experience the experts had but how they thought. You know the famous line that [philosopher] Isaiah Berlin borrowed from a Greek poet, "The fox knows many things, but the hedgehog knows one big thing"? The better forecasters were like Berlin's foxes: self-critical, eclectic thinkers who were willing to update their beliefs when faced with contrary evidence, were doubtful of grand schemes and were rather modest about their predictive ability. The less successful forecasters were like hedgehogs: They tended to have one big, beautiful idea that they loved to stretch, sometimes to the breaking point. They tended to be articulate and very persuasive as to why their idea explained everything. The media often love hedgehogs.
How do you know whether a talking head is a fox or a hedgehog?
Count how often they press the brakes on trains of thought. Foxes often qualify their arguments with "however" and "perhaps," while hedgehogs build up momentum with "moreover" and "all the more so." Foxes are not as entertaining as hedgehogs. But enduring a little tedium is worth it if you want realistic odds on possible futures.