August 16, 2016

Are the many smarter than the few?


I've read the book The Wisdom of Crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations written by James Surowiecki. The title is actually a small lie because we will see that in some cases the many are not always smarter than the few, such as when crazy speculators are investing in overpriced Internet stocks. But the idea of the wisdom of crowds is not that a group will always give you the right answer but that on average it will consistently come up with a better answer than any individual could provide.

First of all I'm personally a fan of the idea of using the crowd. One of my favorite sites is a social news site called Hacker News. The basic idea behind that site is that the users are searching through the Internet for news and then they submit what they think are the best news to the site. Then the users of the site are voting on what they think are the best news. The result is much better compared with the traditional online news paper where just a few journalists are searching through the Internet and then they publish what they think the readers want (even though they sometimes write their own articles, but most of it is copy-and-paste from other sources). It's much better if the readers are choosing what they want because they know what they want.

But after reading The Wisdom of Crowd I realized that there's one small problem with Hacker News. The basic idea behind social news sites is that you submit link to something you like, and then the link ends up in the new-section. Now the problem is that most users are voting on links other users already voted on. Far from all users are visiting the new-section because they don't have the time to go through all the useless links, so they are just visiting the best-section, which is the section where the links with the most votes are displayed. So the result is that some links that a lot of users would have liked end up with no votes because everyone is not voting on the new links independently. A better way could be to hide new links among the best links?

Why is a site like Hacker News (or Reddit which is another similar site) working, while journalist, who are supposed to do the same job, are failing? The basic idea from the book is that:
"Even if most of the people within a group are not especially well-informed or rational, it can still reach a collectively wise decision." 
One example from the book is judging how many balls there's in a transparent jar. It turned out that no single person could make a better guess than the average guess of the group. In some experiments, some individuals could make a better guess than the group, but they were most likely just lucky.

So chasing an expert is often a mistake. What you should do is to look at the group's decision. But this is not always the case because groups work well under certain circumstances, and less well under other:
  • Groups need rules to maintain order and coherence.
  • Groups benefit from members talking to and learning from each other. But the members of the group should make the "votes" independently of the other members. Independence doesn't mean in isolation from everyone else, but relative freedom from the influence of others.  
  • Groups can sometimes be big and sometimes small. But small groups risk having too little diversity (not enough different views) and large groups can be unmanageable. 

According to the book, diversity and independence are important because the best collective decisions are the product of disagreement and contest - not consensus or compromise. This is why we sometimes see stock market bubbles, because the investors (or rather speculators) are not making the decision to buy the stocks independently of everyone else. They see that someone has made a million from ABC.com and now they also want to make a million by buying the same company. Neither has this group any diversity in it because everyone is just buying, while those who have the opposite view are ignored.

One example where a diverse group is making the "votes" independently is Google's algorithm. When I wrote the links in this article, I didn't care about what anyone else thought about to which sites I should link. Google will then use these links to determine if the sites to which I link should show up higher in the search results when someone is searching for the book The Wisdom of Crowds. Google has rules that determine which sites should be more important than other sites and the entire Internet is considered a big group, which can be unmanageable, but Google has figured out how to make the Internet more manageable. And since Google is the world's largest search engine, the ideas behind this book are actually working: under the correct circumstances the many are smarter than the few.    

No comments:

Post a Comment