spaCy for lemmatization

spaCy only supports lemmatization. As discussed by spaCy creator Matt Honnibal in issue #327 on GitHub, stemmers are rarely a good idea.

We want to treat meet/NOUN differently from meeting/VERB. Unlike Stanford NLTK, which was created to teach and introduce as many NLP ideas as possible, spaCy takes an opinionated stand against stemming.

spaCy does lemmatization for you by default when you process the text with the nlp object. This information is stored in the lemma attribute for each token. spaCy stores the internal hash or identifier, which spaCy stores in token.lemmaThis numerical hash has no meaning for us. This numerical representation helps spaCy access and manipulate information much faster than its other ...

Get Natural Language Processing with Python Quick Start Guide now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.