Collective Intelligence in Action

There's a great deal of wisdom in a crowd, but how do you listen to a thousand people talking at once? Identifying the wants, needs, and knowledge of internet users can be like listening to a mob.

In the Web 2.0 era, leveraging the collective power of user contributions, interactions, and feedback is the key to market dominance. A new category of powerful programming techniques lets you discover the patterns, inter-relationships, and individual profiles—the collective intelligence—locked in the data people leave behind as they surf websites, post blogs, and interact with other users.

Collective Intelligence in Action is a hands-on guidebook for implementing collective-intelligence concepts using Java. It is the first Java-based book to emphasize the underlying algorithms and technical implementation of vital data gathering and mining techniques like analyzing trends, discovering relationships, and making predictions. It provides a pragmatic approach to personalization by combining content-based analysis with collaborative approaches.

Following a running example in which you harvest and use information from blogs, you learn to develop software that you can embed in your own applications. The code examples are immediately reusable and give the Java developer a working collective intelligence toolkit.

Along the way, you work with a number of APIs and open-source toolkits including text analysis and search using Lucene, web-crawling using Nutch, and applying machine learning algorithms using WEKA and the Java Data Mining (JDM) standard.

• Architecture for embedding intelligence in your application
• Developing metadata about the user and content
• Gather intelligence from tagging and build tag clouds
• Introduction to intelligent web crawling and Nutch
• Harvesting information from the blogosphere
• Build a text analysis toolkit leveraging Lucene
• Business intelligence and data mining for recommendations and promotions
• Leveraging open-source data mining toolkit WEKA and the Java Data Mining (JDM) standard
• Incorporating intelligent search in your application
• Building a recommendation engine—finding related users and content
• Real-world case studies of Amazon, Google News, and Netflix personalization.

This book assumes you have a basic level of Java coding skills.

Satnam Alag, PhD, is currently the Vice President of Engineering at NextBio, a vertical search engine and a Web 2.0 collaboration application for the life sciences community. He is a seasoned software professional with over fifteen years of experience in machine learning and over a decade of experience in commercial software development and management. Dr. Alag worked as a consultant with Johnson & Johnson's BabyCenter where he helped develop their personalization engine. Prior to that he was the Chief Software Architect at Rearden Commerce and began his career at GE R&D. He is a Sun Certified Enterprise Architect (SCEA) for the Java Platform. Dr. Alag earned his PhD in engineering from UC Berkeley and his dissertation was in the area of probabilistic reasoning and machine learning. He has published numerous peer-reviewed articles.