Review of The Signal and the Noise

I recently finished reading Nate Silver's The Signal and the Noise, why so many predictions fail - but some don't. This book really opened my eyes to Bayesian inference, i.e., the ability to make successively better predictions by incorporating estimates based on prior knowledge. Beyond that, I found Silver's study of several fields where prediction are commonly used to be insightful. He looks at sports gambling, economics, political elections, flu epidemics, weather, and military and terrorist attacks. Although his own expertise is in baseball and election predictions, where he has a great track record, Silver's analysis transcends the statistics to include a study of human nature and of technology.

In the field of climate forecasts, the author shows how the commercial broadcast company weather forecasters use a different standard for determining what to report than do the national weather forecasters who are more independent. This leads to weather forecasts that are biased toward more chance of precipitation because people are more likely to enjoy an unexpected sunny day than an unexpected rain shower.

Something I'm finding more and more often in reading about how we think and process information is that there is a human tendency to see a pattern where there is none. Silver makes this point in regard to the much larger volumes of information that are available to us in many fields as a result of technological progress. More information leads to more theories. But it has not necessarily led to better predictions.

Silver's insight about the explosion of information following the invention of the printing press leading to increased sectarianism is brilliant. With more information available to people, via books and pamphlets, those with strong beliefs were able to publish their stories and rationales in a form that presented them as the truth. This led to more divisiveness. The explosion of information as resulting from the internet's wide usage is likely leading to a similar divisiveness in political opinion.

The difference between risk and uncertainty is something Silver does a nice job explaining. He says that risk "is something that you can put a price on..." while uncertainty is "risk that is hard to measure." How do we deal with uncertainty? Silver puts his money behind the concept of Bayesian inference whereby we reduce uncertainty in a gradual manner, based on our prior experience, adding our knowledge from new experiences to that prior experience.

The emergence of "complex systems" in our lives has led to some interesting mistakes in prediction. The weather, or the climate if you think longer-term, is a complex system with many variables. The economy is another. Silver delivers a scathing critique of the major stock ratings agencies in terms of how they miscalculated the risk of a major financial crash. "S&P and Moody's underestimated the default risk associated with CDOs by a factor of two hundred..." reports Silver. The analysis I'll leave for the reader to enjoy. Just one additional point, though, that he makes is that the gap between what we know and what we think we know is increasing as the volume of information available increases. That's a caveat emptor for the predictors if I've ever heard one!

Silver does a nice job of explaining two different "personas" in terms of experts making predictions. He draws from a study done by Philip Tetlock, a psychology and political science professor, who, while studying economists' predictions, decided to test the economists using some of his psychological profile tests. His study eventually covered other experts where prediction was performed, and spanned fifteen years. What he found was that the experts fell into one of two groups: hedgehogs or foxes. Hedgehogs were more convinced their theory was correct and more likely to not change it based on new information. Foxes were just the opposite. What he found was that foxes were more likely to make better predictions. This whole area of study, looking at how one's psychological profile affected one's ability to use information to make predictions, is exciting and is actually something that can be applied, with caution, across a lot of disciplines. Even in software development, where I work, I can see how it can apply.

Silver's analysis of the Pearl Harbor and 9/11 attacks bears some mention. In both cases, he explains how both events were, to some extent, statistically likely, but that both types of attacks were not thought probablye because they were unfamiliar. The United States in late 1941 was on alert for industrial espionage both in Hawaii and on the mainland, because it was thought Japanese Americans or Nazi sympathizers were likely to strike in that way. In the Pacific, Japanese attacks on southeast Asian nations was considered a high probability given that there was a lot of Japanese radio communication in those areas. In 2001, there had never been a serious airplane attack against a building. If an airplane were to be hi-jacked, it was thought to be with the goal of taking the plane to foreign soil.

Finally, Silver does a nice job of explaining power-law distributions, over-fitting a model and Bayesian inference. It encouraged me to want to learn more about the mathematics. He also has an incredible number of notes and references in the book. I found myself reading many footnotes, which had interesting commentary.

Overall, this was, in my opinion, a ground-breaking book, at least for the lay reader if not for a professional commonly involved in providing predictions.

Occupy, and Cause and Effect

The intensity of the Occupy movement has been growing week by week, reaching a crescendo this past week with the apparent coordinated shutdowns by US cities and then the backlash, followed by unwarranted pepper-sprayings, followed by viral postings of those incidents. A local Portlander who I follow on Twitter has been a regular at OccupyPortland and I noticed that he suggested that people should follow @_Capitalism_. I read through several pages worth of @_Capitalism_'s tweets and they are a witty and potentially sobering expose of the ills of capitalism as currently practiced.

This got me thinking more deeply about the larger impact of the Occupy movement. The Great Recession (that we are either in the tail of or which recently passed, depending on which economist you read) has occurred at a time when many college graduates are having difficulty finding jobs. The bailout of financial institutions followed by publication of bonus plans for financial executives have also occurred at the same time. On the surface, these two factors: a very sluggish job market and the excessive bonuses and bailouts certainly justify the indignation witnessed in the Occupy movement. If you add to those factors the polarization of Washington politics, we are clearing in a defining moment of time.

I have always believed that technological innovation would drive skills enhancements for many jobs. There would be less of a need for highly repetitive, mechanistic work, and more need for thoughtful analysis. A good example is the work of a nurse where the ability to read and respond to digital monitoring of a patient's condition has become routine. There is more expected of us at work now that we have automated some of the routine calculations and steps of a task. It is also true that some of the finer mechanical skills that technicians applied throughout the latter half of the nineteenth and the entire twentieth century have become less in demand as those workers have been replaced by automation.

It's difficult enough determining how best to use new technology on the job, more so to figure out the right mix of human and machine resources for an economy as a whole. It is not a zero-sum game where more technology at work means less humans at work. As computers started monitoring patients, or assembly-line operations, the skills required of people to work with that monitoring equipment changed. New industries were created to produce innovative gadgets and peripheral devices, and of course, software. Back when computers were starting to appear on office desks, who could have predicted that we would be funding companies to create social media applications for mobile phones a generation or so later?

There are clear indications that we need to continually learn how to use and shape technology if we are to be gainfully employed in occupations that challenge us and have meaning to us. As exponential growth in some factors of technology continue, it will be critical for us to keep up.

In their book, Race Against the Machine, Brynjolfsson and McAfee refer to the second half of the chessboard, a term coined by Ray Kurzweil to indicate a point where "an exponentially growing factor begins to have a significant economic impact." As the authors put it:

"Kurzweil’s point is that constant doubling, reflecting exponential growth, is deceptive because it is initially unremarkable. Exponential increases initially look a lot like standard linear ones, but they’re not. As time goes by—as we move into the second half of the chessboard—exponential growth confounds our intuition and expectations. It accelerates far past linear growth..."

As evidence of the fact that we are truly experiencing technological change that confounds us, they refer to Google's automobile driving itself, IBM's Watson consistently winning at Jeopardy and GeoFluent's ability to do real-time translation of online chat messages.

A partial answer for the extremely sluggish job growth we've seen over the past year is, I believe, the technological unemployment resulting from this exponential growth. Technological innovation is seen in many traditional industries, and in fact is responsible, according to Brynjolfsson and McAfee, for separating some of the leading companies from their competitors.

As for the increasing income disparity between the 99% and the 1%, exponential technological growth will only exacerbate this. Owners of capital, which includes corporate ownership of technology, have reaped greater profits from the explosive productivity of these non-human resources. Unless the majority of workers (by which I mean workers without significant ownership in an enterprise) can sell their skill at a rate that keeps up with the growth of technological capital, then workers will necessarily earn a smaller piece of the pie.

As a student of economics, this is how I see what is happening in our current crisis. It is not a moral judgement by any means, but represents what appears, to me, to be happening. As an experienced software engineer, I see every day who competes and am aware of those that do not, and without self-training and pro-active behavior, I see the crisis getting worse for most people.

So, it is not purely the mis-administration of public funds to prop up reckless financiers, nor the greed of the 1% that is necessarily responsible for our current economic climate. There are longer-term forces at work which require not only good stewardship from our corporate and government leaders but good stewardship by each and every person of their own career.

I started out this post by addressing the Occupy movement and I hope to have shed some light on the longer-term forces that are important when looking at cause and effect. Long-term technological progress does not explain the reckless behavior of Wall Street firms trying to make money on securities based on bad mortgages. But it does explain at least part of the increasing disparity in income we've seen in our society over the past twenty or thirty years. In the spirit of cooperation, I wonder if we, as citizens, might be able to find ways to help ourselves become better prepared for a more technologically-oriented future, while at the same time using our newly-acquired skills to monitor and police our society so that irresponsible behavior in the so-called white-collar world is more transparent and more easily deterred.

Leaving this open-ended, without answer at this point gives me an opportunity to think and write more on these possibilities. I'd love to hear your thoughts.

Using Social Media to Enhance Community

I'm doing a new community film project entitled 'Using Social Media to Enhance Community' (until I can come up with a better name). This film will briefly explore what social media is (e.g., Facebook, Twitter, blogging, wikis) followed by interview clips, short screencasts, and some fly-on-the-wall views of at least one in-person meeting of people who are involved in social media.

Right now I'm looking for some non-technical people who would provide a balance to the tech-savvy people who I am interviewing. The focus for the interview and video with non-technical people is to explore how their reaction to using social media applications for community-building. Although the tone of this film is intended to be positive toward the use of social media for communities, hearing some of the difficulties would also be constructive.

I may also explore how social media is used in secondary or pre-secondary education, how it is perceived and used by the leaders and workers of the future, although I haven't decided if this will make it into this film.

If you are someone interested in participating in this project, to be interviewed or to recommend others, please contact me.

Social Networking, Social Business, and the Future

I'm struggling with how to take advantage as well as how to assess the utility of social networking sites and tools like Facebook, Twitter, Bebo, MySpace and even StumbleUpon. There are some obvious positive scenarios as evidenced by Beth Kanter's (and others') use of these networks to drive a fund-raising challenge like America's Giving Challenge. It is absolutely awesome that the Sharing Foundation, supporting poor children in Cambodia, was able to top the leadership chart of the forementioned Challenge mostly by the sweat and tears Beth had shed over the last few days.

But as this article points out, there has been an apparent drop in the usage of these networks and they are still looking for a raison d'etre while most of the users appear to be simply hanging out and sharing what they're currently doing.

I'm optimistic that the social networking sites represent an evolution in human interaction. We've gone from 99% of the population knowing only people living within a few miles to a world where we can instantly make friends around the world, talk to them and share one's lives, using commodity technology. All within the past couple of hundred years! It's way too early to judge the utility of the current Web 2.0 technologies except to say they're sexy, cool, fun and, for a small number of hard-working people, lucrative.

But I want to take this post in a slightly different direction. There are major developments occurring right now in technology, environmental and social systems that will have impacts on every person on this planet. The Web 2.0 phenomena is one part of the technology surge. There are also all of the creative uses of cell phone technology (for improving business communications, e.g.) and applied approaches to nanotechnology to solve problems.

In the social sphere, there is a surge of interest in eradicating extreme poverty. And one aspect of this is reflected on by Dave Richards on his blog as he breaks down what a social business is. I feel that there's a synergy between the technological developments and the social developments (whether they be the social networking web sites or social business models). Bringing these developments to fruition to solve extreme poverty is where I think we, as a unified group of people on this planet, need to be.

Wired Reading - What Fun!

With Maria studying in a paralegal program now, we have had less time together. Tonight we got out for a great budget meal at Laughing Planet and followed that with a visit to Barnes & Noble where Maria picked up some reading material (where she gets the time for leisure reading I don't know). We decided to head over to Vivace, one of our favorite coffee shops to read. I brought along a novel by James Rollins called Black Order. But after sitting down, I brought out my new phone, a TMobile Dash, aka the HTC Excalibur and browsed over to my Google Reader where I read some interesting blog entries, most notably one by Adrian Holovaty, a programmer-journalist, a citizen journalist who developed ChicagoCrime.org and EveryBlock


Reading what Adrian is doing, creating mashups that serve local communities, I had a moment where I realized that what I was doing was a completely different take on the passive art of reading. Holding a lightweight phone (with a small but very clear screen by the way) in my hand, I could absorb news or opinions from anywhere in the world instead of limiting myself to the one book, one author of a hardcopy book. Ya this is nothing new but the act of reading took on a more exciting, electric feeling for me.

I was brought a little back to earth when I then tried to write a blog entry about this topic right on the phone. Unfortunately, Blogger required me to send a message to an email address they provide, get back a code and then when I get back to my computer, go to Blogger and give it the code which it sent to my phone. Not exactly a simple write and publish, one-stop process!

Computer Instruction, high-speed access and Cambodia

I'm writing this entry as a response to Beth Kanter's challenge.

What advice would you offer to Mam Sari about incorporating computer instruction on a REALLY slow connection and with one computer connected to the Internet?

If he has time, he can prepare for the class by printing out a Google results page and then annotating with his own comments. For example, he might have (in colored ink) labels for each part of each results item. Going over these as handouts to the class will give them something to study prior to their own practice.

Perform some searches before class and use the forward and back browser buttons to shorten the wait time between pages.

Use the "wait" time effectively. This is where pre-written or printed materials are helpful. Go to Google Help (available through the About Google page) and use some of the pages, printed, as material to expand the students' understanding of what Google offers.


Are there any web resources or books that you think I should send over to him to read?

We in the developed world, have access to high-quality printers, plotters, etc. Having some color charts and handouts would be helpful. I don't know what your budget is, and if the students can read English well, but providing how-to books for the students would be great.


Dream a little dream with me, if we had a fast Internet connection, what are the possibilities?

What do the students need to be successful? Do they need money to go to school? Having access to a high-speed connection gives them the ability to participate in the person-to-person loans that are becoming more common on the Internet (e.g., fynanz). Actually, I'm not sure this requires a fast connection but it would certainly allow students to do a lot more exploration, to see how they can get additional help, even virtual tutoring.

Just today, a friend suggested that it would be cool to have a microfinance-like site similar to Kiva, which instead of providing money for a business, provides money in the form of loans for education. I think it's a great idea, although to get it to work effectively in countries like Cambodia will require some facilitative or management presence in the country. Making sure that the students have the support of their family, who may see education as a drain on their economic needs, to align students with programs and make sure the funds get to the school, etc. Sorry, I may be getting off topic a bit but the more that the everyday person in Cambodia can access the web as an extension of the market, the more likely such ventures will happen.

Community Film Production and Basic Rights Oregon

I'm taking a Field Production class from Portland Community Media and this Saturday we had our first filming session. Our class is divided into two groups, the other group is producing a short film on the pinball craze and our group is producing a film on the effects on families of the injunction on the Oregon Family Fairness Act. On Saturday we visited a family of four in Southeast Portland. Our goals were to get some interesting commentary on the effects on their lives from the two parents and to get some background filler material including their two children.

And wow, were our expectations met and far exceeded! After setting up the Sony DS250 on a tripod, doing the technical preparation (white adjustment, iris setting, audio settings), Kevin, one of our team, sat down next to the camera as the interviewer. The subjects (I don't want to reveal names here in the interest of their privacy, although you will surely be able to see them when the film is aired on local community television) were asked how they met. Let me say, that was all we had to do. These two women gave us such a spirited, interactive, friendly and intelligent story that no other questions were necessary (although we did ask a few others just to give them a break from talking).

I was so entertained by their story! Full of funny stories and poignant moments! Then they naturally gravitated to the subject of the Family Fairness Act and how the current state of the world in Oregon, that they are not officially married, that although they have had two separate wedding or union ceremonies, they are still not legally considered married. The student film team has the tough task of trying to scale this film session down to a short film. Maybe we will be able to convince our instructor, Tim, that this needs to be a longer film.

Before moving on, I can't help but comment on the interview with their two cute daughters who gave us some remarkably nice interview material.

I can't imagine anyone who sees our final film believing that these wonderful people are not a true family and deserve every single right that heterosexual couples receive for their families.

So we learned how to setup equipment, how to film, we did a little roaming camera work to pick up some interesting household views, including some home-schooling, and then we were off. Next Saturday we go to Basic Rights Oregon to do some additional interviewing.