Review of The Signal and the Noise

I recently finished reading Nate Silver's The Signal and the Noise, why so many predictions fail - but some don't. This book really opened my eyes to Bayesian inference, i.e., the ability to make successively better predictions by incorporating estimates based on prior knowledge. Beyond that, I found Silver's study of several fields where prediction are commonly used to be insightful. He looks at sports gambling, economics, political elections, flu epidemics, weather, and military and terrorist attacks. Although his own expertise is in baseball and election predictions, where he has a great track record, Silver's analysis transcends the statistics to include a study of human nature and of technology.

In the field of climate forecasts, the author shows how the commercial broadcast company weather forecasters use a different standard for determining what to report than do the national weather forecasters who are more independent. This leads to weather forecasts that are biased toward more chance of precipitation because people are more likely to enjoy an unexpected sunny day than an unexpected rain shower.

Something I'm finding more and more often in reading about how we think and process information is that there is a human tendency to see a pattern where there is none. Silver makes this point in regard to the much larger volumes of information that are available to us in many fields as a result of technological progress. More information leads to more theories. But it has not necessarily led to better predictions.

Silver's insight about the explosion of information following the invention of the printing press leading to increased sectarianism is brilliant. With more information available to people, via books and pamphlets, those with strong beliefs were able to publish their stories and rationales in a form that presented them as the truth. This led to more divisiveness. The explosion of information as resulting from the internet's wide usage is likely leading to a similar divisiveness in political opinion.

The difference between risk and uncertainty is something Silver does a nice job explaining. He says that risk "is something that you can put a price on..." while uncertainty is "risk that is hard to measure." How do we deal with uncertainty? Silver puts his money behind the concept of Bayesian inference whereby we reduce uncertainty in a gradual manner, based on our prior experience, adding our knowledge from new experiences to that prior experience.

The emergence of "complex systems" in our lives has led to some interesting mistakes in prediction. The weather, or the climate if you think longer-term, is a complex system with many variables. The economy is another. Silver delivers a scathing critique of the major stock ratings agencies in terms of how they miscalculated the risk of a major financial crash. "S&P and Moody's underestimated the default risk associated with CDOs by a factor of two hundred..." reports Silver. The analysis I'll leave for the reader to enjoy. Just one additional point, though, that he makes is that the gap between what we know and what we think we know is increasing as the volume of information available increases. That's a caveat emptor for the predictors if I've ever heard one!

Silver does a nice job of explaining two different "personas" in terms of experts making predictions. He draws from a study done by Philip Tetlock, a psychology and political science professor, who, while studying economists' predictions, decided to test the economists using some of his psychological profile tests. His study eventually covered other experts where prediction was performed, and spanned fifteen years. What he found was that the experts fell into one of two groups: hedgehogs or foxes. Hedgehogs were more convinced their theory was correct and more likely to not change it based on new information. Foxes were just the opposite. What he found was that foxes were more likely to make better predictions. This whole area of study, looking at how one's psychological profile affected one's ability to use information to make predictions, is exciting and is actually something that can be applied, with caution, across a lot of disciplines. Even in software development, where I work, I can see how it can apply.

Silver's analysis of the Pearl Harbor and 9/11 attacks bears some mention. In both cases, he explains how both events were, to some extent, statistically likely, but that both types of attacks were not thought probablye because they were unfamiliar. The United States in late 1941 was on alert for industrial espionage both in Hawaii and on the mainland, because it was thought Japanese Americans or Nazi sympathizers were likely to strike in that way. In the Pacific, Japanese attacks on southeast Asian nations was considered a high probability given that there was a lot of Japanese radio communication in those areas. In 2001, there had never been a serious airplane attack against a building. If an airplane were to be hi-jacked, it was thought to be with the goal of taking the plane to foreign soil.

Finally, Silver does a nice job of explaining power-law distributions, over-fitting a model and Bayesian inference. It encouraged me to want to learn more about the mathematics. He also has an incredible number of notes and references in the book. I found myself reading many footnotes, which had interesting commentary.

Overall, this was, in my opinion, a ground-breaking book, at least for the lay reader if not for a professional commonly involved in providing predictions.

Occupy, and Cause and Effect

The intensity of the Occupy movement has been growing week by week, reaching a crescendo this past week with the apparent coordinated shutdowns by US cities and then the backlash, followed by unwarranted pepper-sprayings, followed by viral postings of those incidents. A local Portlander who I follow on Twitter has been a regular at OccupyPortland and I noticed that he suggested that people should follow @_Capitalism_. I read through several pages worth of @_Capitalism_'s tweets and they are a witty and potentially sobering expose of the ills of capitalism as currently practiced.

This got me thinking more deeply about the larger impact of the Occupy movement. The Great Recession (that we are either in the tail of or which recently passed, depending on which economist you read) has occurred at a time when many college graduates are having difficulty finding jobs. The bailout of financial institutions followed by publication of bonus plans for financial executives have also occurred at the same time. On the surface, these two factors: a very sluggish job market and the excessive bonuses and bailouts certainly justify the indignation witnessed in the Occupy movement. If you add to those factors the polarization of Washington politics, we are clearing in a defining moment of time.

I have always believed that technological innovation would drive skills enhancements for many jobs. There would be less of a need for highly repetitive, mechanistic work, and more need for thoughtful analysis. A good example is the work of a nurse where the ability to read and respond to digital monitoring of a patient's condition has become routine. There is more expected of us at work now that we have automated some of the routine calculations and steps of a task. It is also true that some of the finer mechanical skills that technicians applied throughout the latter half of the nineteenth and the entire twentieth century have become less in demand as those workers have been replaced by automation.

It's difficult enough determining how best to use new technology on the job, more so to figure out the right mix of human and machine resources for an economy as a whole. It is not a zero-sum game where more technology at work means less humans at work. As computers started monitoring patients, or assembly-line operations, the skills required of people to work with that monitoring equipment changed. New industries were created to produce innovative gadgets and peripheral devices, and of course, software. Back when computers were starting to appear on office desks, who could have predicted that we would be funding companies to create social media applications for mobile phones a generation or so later?

There are clear indications that we need to continually learn how to use and shape technology if we are to be gainfully employed in occupations that challenge us and have meaning to us. As exponential growth in some factors of technology continue, it will be critical for us to keep up.

In their book, Race Against the Machine, Brynjolfsson and McAfee refer to the second half of the chessboard, a term coined by Ray Kurzweil to indicate a point where "an exponentially growing factor begins to have a significant economic impact." As the authors put it:

"Kurzweil’s point is that constant doubling, reflecting exponential growth, is deceptive because it is initially unremarkable. Exponential increases initially look a lot like standard linear ones, but they’re not. As time goes by—as we move into the second half of the chessboard—exponential growth confounds our intuition and expectations. It accelerates far past linear growth..."

As evidence of the fact that we are truly experiencing technological change that confounds us, they refer to Google's automobile driving itself, IBM's Watson consistently winning at Jeopardy and GeoFluent's ability to do real-time translation of online chat messages.

A partial answer for the extremely sluggish job growth we've seen over the past year is, I believe, the technological unemployment resulting from this exponential growth. Technological innovation is seen in many traditional industries, and in fact is responsible, according to Brynjolfsson and McAfee, for separating some of the leading companies from their competitors.

As for the increasing income disparity between the 99% and the 1%, exponential technological growth will only exacerbate this. Owners of capital, which includes corporate ownership of technology, have reaped greater profits from the explosive productivity of these non-human resources. Unless the majority of workers (by which I mean workers without significant ownership in an enterprise) can sell their skill at a rate that keeps up with the growth of technological capital, then workers will necessarily earn a smaller piece of the pie.

As a student of economics, this is how I see what is happening in our current crisis. It is not a moral judgement by any means, but represents what appears, to me, to be happening. As an experienced software engineer, I see every day who competes and am aware of those that do not, and without self-training and pro-active behavior, I see the crisis getting worse for most people.

So, it is not purely the mis-administration of public funds to prop up reckless financiers, nor the greed of the 1% that is necessarily responsible for our current economic climate. There are longer-term forces at work which require not only good stewardship from our corporate and government leaders but good stewardship by each and every person of their own career.

I started out this post by addressing the Occupy movement and I hope to have shed some light on the longer-term forces that are important when looking at cause and effect. Long-term technological progress does not explain the reckless behavior of Wall Street firms trying to make money on securities based on bad mortgages. But it does explain at least part of the increasing disparity in income we've seen in our society over the past twenty or thirty years. In the spirit of cooperation, I wonder if we, as citizens, might be able to find ways to help ourselves become better prepared for a more technologically-oriented future, while at the same time using our newly-acquired skills to monitor and police our society so that irresponsible behavior in the so-called white-collar world is more transparent and more easily deterred.

Leaving this open-ended, without answer at this point gives me an opportunity to think and write more on these possibilities. I'd love to hear your thoughts.