Raspberry Pi 3 Cookbook for Python Programmers - Third Edition

How to do it...

1. Import the following packages:

from nltk.tokenize import RegexpTokenizerfrom nltk.stem.snowball import SnowballStemmerfrom gensim import models, corporafrom nltk.corpus import stopwords

2. Load the input data:

def load_words(in_file):  element = []  with open(in_file, 'r') as f:    for line in f.readlines():      element.append(line[:-1])  return element

3. Class to pre-process text:

classPreprocedure(object):  def __init__(self):    # Create a regular expression tokenizer    self.tokenizer = RegexpTokenizer(r'w+')

4. Obtain a list of stop words to terminate the program execution:

    self.english_stop_words= stopwords.words('english')

5. Create a Snowball stemmer:

    self.snowball_stemmer = SnowballStemmer('english')  

6. Define a function to perform ...