Book review: Superforecasting: The art and science of prediction. Crown Publishers, New York, NY. Tetlock. E. Philip, Gardner, Dan (2015)

Abstract

There are many things that are good about this book. Philip E Tetlock is a scholar with an impressive number of publications and citation. The book is well-written and easy to read, but that is also the best that can be said.

The book falls into a long line of bestselling books that have an extravagantly attractive title that has little to do with the content, and a first chapter that is all about promises of what is to be delivered in the following pages. As such, this is all too common in the management literature in general as we have known it since the early 1980s, maybe even earlier. It throws around the names of famous people and stories people can relate to. But what is the problem with that, the reader may ask. Well the problem is that these types of management books continue to have a significant influence on practice, much more so than scientific articles or more instrumental books on intelligence analysis.  This is not a new phenomenon either but has been going on since “In search of Excellence” or maybe even longer. For the most part though these books are being discredited in the long run, but then it is too late, as their content has already been put into practice.

For one thing there is nothing that has been presented in the book that helps explain why the project was better at predicting events than anybody else, if we are to believe that that is true. More worryingly, the book does not say how the authors and the project beat the other analysts, if it was by simply using a more vague language in its estimates or by the way correct answers were calculated. The rules of these competitions are never explained, at least not in the book.

The main idea in the book is that if you give precise questions and ask for answers expressed in numbers for specific time frames, then you can also sit back and wait to measure the results. You will then know how good you are. That by itself is not a bad idea. Instead we are led on a series of loose threads and assumptions, by the authors who are expert analysts because they did so – “it took years” - and won. It seems like a proven way to sell consultancy, but does not convince a reader who is even half awake.

Clearly psychology is important for decision making and forecasting, especially when confronted with social situations where an outcome is the result of the interaction and the expectations of several individuals with different interests and values. Some of these problems can be modelled using game theory, but the authors fail to see that this is only one half of the equation. The other half is what you actually know. The intelligence reality of Mr Tetlock is much like that of a psychologist in a poker game. He does not know what the other person knows but tries to guess it based on his behavior. That is a much riskier way of solving a problem than using resources to actually find out. Good intelligence is about finding out what hand was actually dealt. This will give us certainty to know how we could win the game, or at least avoid losing more money than what was in the pot. Psychology is important in knowing how the player will behave. It is this other part of the equation—that the psychological insights are valuable—that Tetlock introduces in this book.

It’s a good suggestion to test or check guesses to learn from them, but it’s hardly a new or novel idea. It’s true that it is “astonishing” how many organizations do not check the intelligence they produce or buy, but it’s hardly a new problem or even surprising.

The book is one in a long tradition of “hype” books which are so popular and not only in the Anglo-Saxon world, similar to Nassim Taleb’s book “Black Swan”, which the authors also refer to. You take something that is merely common sense and present it in an appealing way, such as that complete unknowns are like black swans. The reader will not have learned anything new, but old wisdom is frightfully well packaged, thus appealing. It does not help that the authors disagree with Taleb in that they think that many swans that people say are black are in fact grey (another metaphor of the same type).

I said at the beginning that this is a good book. The reason for this is that it contains many good rules of thumb. Unfortunately, they are not listed in any single place in the book. We should break large questions into many small questions. We should make scorekeeping an integral part of intelligence analysis (p. 259). That is a simple but important lesson. Thus the book is worth reading.