NLP Terminologies

Before diving deeper into text mining techniques, it helps to be familiar with the core vocabulary used throughout Natural Language Processing.

Key Terms

• Corpus - a large, structured collection of text used for analysis or training
• Tokens - the individual units (usually words) that text is split into during tokenization
• Stop words - common words like "the", "is", and "and" that are often removed since they carry little meaning
• Stemming - reducing a word to its root form by chopping off suffixes, e.g. "running" → "run"
• Lemmatization - reducing a word to its dictionary base form using vocabulary and grammar rules, e.g. "better" → "good"
• POS tagging - labeling each token with its part of speech (noun, verb, adjective, etc.)
• N-grams - contiguous sequences of N words, used to capture some context beyond single words

Stemming vs Lemmatization in Python

from nltk.stem import PorterStemmer, WordNetLemmatizer

stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()

print(stemmer.stem("running"))       # run
print(lemmatizer.lemmatize("better", pos="a"))  # good