O'Reilly logo

Python 3 Text Processing with NLTK 3 Cookbook by Jacob Perkins

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Distributed word scoring with Redis and execnet

We can use Redis and execnet together to do distributed word scoring. In the Calculating high information words recipe in Chapter 7, Text Classification, we calculated the information gain of each word in the movie_reviews corpus using a FreqDist and ConditionalFreqDist. Now that we have Redis, we can do the same thing using a RedisHashFreqDist and a RedisConditionalHashFreqDist, and then store the scores in a RedisOrderedDict. We can use execnet to distribute the counting in order to get a better performance out of Redis.

Getting ready

Redis, redis-py, and execnet must be installed, and an instance of redis-server must be running on localhost.

How to do it...

We start by getting a list of (label, words) ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required