Raspberry Pi 3 Cookbook for Python Programmers - Third Edition

How to do it...

1. Introduce sentence tokenization:

from nltk.tokenize import sent_tokenize

2. Form a new text tokenizer:

tokenize_list_sent = sent_tokenize(text)print "nSentence tokenizer:" 
print tokenize_list_sent 

3. Form a new word tokenizer:

from nltk.tokenize import word_tokenize 
print "nWord tokenizer:" 
print word_tokenize(text) 

4. Introduce a new WordPunct tokenizer:

from nltk.tokenize import WordPunctTokenizer 
word_punct_tokenizer = WordPunctTokenizer() 
print "nWord punct tokenizer:" 
print word_punct_tokenizer.tokenize(text) 

The result obtained by the tokenizer is shown here. It divides a sentence into word groups: