⬡ Hub
Skip to content

NLTK: Tokenization, Stemming, and Lemmatization

Text preprocessing is a crucial step in Natural Language Processing (NLP) workflows. NLTK (Natural Language Toolkit) provides powerful tools for these fundamental tasks, helping to transform raw text into a format suitable for analysis or machine learning models.

1. Tokenization

Tokenization is the process of breaking down a stream of text into smaller units called "tokens". These tokens can be words, phrases, symbols, or other meaningful elements. NLTK offers various tokenizers.

a. Word Tokenization

Splitting text into individual words.

import nltk
from nltk.tokenize import word_tokenize

# Download 'punkt' tokenizer models if not already downloaded
try:
    nltk.data.find('tokenizers/punkt')
except LookupError:
    nltk.download('punkt')

text = "NLTK is a powerful library. It helps with natural language processing tasks efficiently. Don't you agree?"

# Tokenize the text into words
tokens_words = word_tokenize(text)
print("Word Tokens:", tokens_words)

b. Sentence Tokenization

Splitting a paragraph or document into individual sentences.

import nltk
from nltk.tokenize import sent_tokenize

text = "Hello, world! NLTK is awesome. How are you doing today? It makes NLP easier."

# Tokenize the text into sentences
tokens_sentences = sent_tokenize(text)
print("Sentence Tokens:", tokens_sentences)

2. Stemming

Stemming is the process of reducing inflected (or sometimes derived) words to their word stem, base, or root form. The stem is not necessarily a grammatically correct word.

a. Porter Stemmer

One of the most widely used stemmers, known for its simplicity and effectiveness.

import nltk
from nltk.stem import PorterStemmer

stemmer = PorterStemmer()

words_to_stem = ["running", "runs", "runner", "easily", "fairly", "connection", "connections"]
stemmed_words = [stemmer.stem(word) for word in words_to_stem]
print("Porter Stemmer results:", stemmed_words)

b. Lancaster Stemmer

A more aggressive stemmer compared to Porter, often producing shorter stems.

import nltk
from nltk.stem import LancasterStemmer

stemmer_lancaster = LancasterStemmer()

words_to_stem = ["running", "runs", "runner", "easily", "fairly", "connection", "connections"]
stemmed_words_lancaster = [stemmer_lancaster.stem(word) for word in words_to_stem]
print("Lancaster Stemmer results:", stemmed_words_lancaster)

3. Lemmatization

Lemmatization is the process of grouping together the inflected forms of a word so they can be analyzed as a single item, identified by the word's lemma, or dictionary form. Unlike stemming, lemmatization considers the context and converts the word to its meaningful base form, which is a valid word in the language.

a. WordNet Lemmatizer

NLTK's WordNetLemmatizer uses WordNet's built-in dictionary to look up lemmas. It requires the 'wordnet' corpus.

import nltk
from nltk.stem import WordNetLemmatizer

# Download 'wordnet' corpus if not already downloaded
try:
    nltk.data.find('corpora/wordnet')
except LookupError:
    nltk.download('wordnet')

lemmatizer = WordNetLemmatizer()

words_to_lemmatize = ["running", "runs", "ran", "better", "best", "cars", "geese", "mice"]

# Lemmatize with default Part-of-Speech (POS) tag (noun)
lemmas_default = [lemmatizer.lemmatize(word) for word in words_to_lemmatize]
print("Lemmas (default POS):", lemmas_default)

# Lemmatize with specified POS tag (e.g., 'v' for verb, 'a' for adjective)
print("Lemmatizing 'running' as verb:", lemmatizer.lemmatize("running", pos="v"))
print("Lemmatizing 'better' as adjective:", lemmatizer.lemmatize("better", pos="a"))
print("Lemmatizing 'ran' as verb:", lemmatizer.lemmatize("ran", pos="v"))
print("Lemmatizing 'is' as verb:", lemmatizer.lemmatize("is", pos="v"))

4. Stop Words

Stop words are common words (like "the", "a", "is", "in") that often carry little significant meaning and can be removed during preprocessing to reduce noise and computational load, especially in tasks like text classification or information retrieval.

import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

# Download 'stopwords' corpus if not already downloaded
try:
    nltk.data.find('corpora/stopwords')
except LookupError:
    nltk.download('stopwords')

stop_words = set(stopwords.words('english'))
print("First 10 English stop words:", list(stop_words)[:10])

example_text = "This is an example sentence, demonstrating the removal of stop words."
word_tokens = word_tokenize(example_text)

filtered_words = [word for word in word_tokens if word.casefold() not in stop_words and word.isalnum()]
print("\nOriginal words:", word_tokens)
print("Filtered words (stopwords removed):", filtered_words)

Further Topics:

  • Part-of-Speech (POS) Tagging
  • Named Entity Recognition (NER)
  • Chunking and Chinking
  • Parsing
  • WordNet
  • Text Classification with NLTK features

These basic preprocessing steps are foundational for almost any NLP task, from sentiment analysis to machine translation. Choosing the right technique (e.g., stemming vs. lemmatization) depends on the specific requirements of your application.