"Thinking, fast and slow" by Daniel Kahneman

This book is a sort of retrospective of the life's work of Daniel Kahneman, a psychologist who won the Nobel Prize for Economics by replacing the rational agent of classical Economics with a real person.

It seeks to divide human thinking into two systems; there are three ways he does this. In part one he compares the fast, intuitive ways in which we think with the slow, effort-full, rational ways of thinking. Because rational thinking is hard work we lazt humans tend to default to intuition which makes us more gullible. In part two he compares the rational agent of Economics (the 'Econ') with Humans. Part three pits the present against the past and shows how what we remember about what we experienced is rarely the same as what we experience.

Kahneman recounts the hundreds of experiments he has conducted during his long career. Many of them offer profound insights into how humans operate. It is clear that if one wishes to improve communications, improve pedagogy, or manipulate people better, there are lessons to be learnt.

One little quibble: many of these experiments were conducted with someone called Amos. Because I had skipped the introduction (I often read introductions at the end because I believe that if a book is strong enough it should stand without the introduction) I did not know who he was talking about. It was Amos Tversky, a long time collaborator, now dead. In some ways this book is Kahneman's tribute to Mr Tversky.

One thing I loved was the evidence base. Many of these experiments prove counter-intuitive conclusions. This hammers home the fact that we are not who we think we are. This book has a massive evidence base for Kahneman's view of humans, in contrast to the scarcely visible evidence base of Roger Scruton's The Uses of Pessimism.

A lot of what Kahneman has to say I have encountered before, for example in Nudge which was written by Richard Thaler and Cass Sunnstein, both former colleagues of Kahneman. For example, I knew about anchoring and that algorithms give better long-range forecasts than expert opinions. Nevertheless, this is a brilliant introduction to these ideas  if you are new to the topic and possibly the most comprehensive text I have encountered (but also try Irrationality by Stuart Sutherland). My only real quibble is that Kahneman does not go into sufficient detail about Bayes Theorem (for which you need The Signal and the Noise by Nate Silver).

Not only is this a book full of fascinating ideas but it is also extrememly readable. August 2013; 418 pages.

"The signal and the noise" by Nate Silver

Nate is the man who called 49 out of 50 states correctly in the 2008 US Presidential election; he has earned money from online poker as well as by setting up a baseball stats prediction website. It was therefore refreshing to find that he was so sceptical about the prediction industry.

His concerns are:

• Human forecasters are unreliable because:
• As humans we are prone to attending best to data that confirms our prejudices. 
• Sometimes there are incentives to be biassed (people prefer weather forecasters who predict rain when it turns out sunny rather than vice versa so there is a perverse incentive to over-forecast rain)
• As humans we like spotting patterns even in random noise
• Many of use are 'hedgehogs' who specialise in one very big idea (rather than 'foxes' who generalise in lots of ideas) and we seek to tell 'stories' that use data when it adds to the narrative and explains it away when it doesn't
• We're rubbish at estimating probabilities: in particular, we convert small probabilities into impossibilities and large probabilities into certainties
• Most patterns in data are swamped by noise
• Things such as earthquakes and stock markets follow a power law (probably because of feedback effects)  which makes statistical forecasting possible but exact prediction impossible: we know the chance of a big earthquake striking LA but we don't know when it will happen
• You can only make a decent living from poker (or the stock market) while there are enough bad players/ investors in the pool whom you can exploit. As they drop out it becomes harder and harder to have an edge. 
• Big data crunching methods ('data-mining') are likely to find patterns because if you have enough variables all wobbling randomly then for some time at least two of them will correlate. So the patterns data-mining finds are quite possibly false positives. 

This means that you need to start with a theory before looking at the data.

This means that Bayesian logic is the way to go to detect patterns. First establish your prior probabilistic expectations, then run the experiment or check the data and then refine your probabilities.

This was a good book though it didn't live up to my expectations. It was easy to read but I was a bit bored by the obsession with baseball.

I certainly learnt about Bayes! March 2013; 454 pages