⬡ Hub
Skip to content

NLTK: Natural Language Toolkit

The Natural Language Toolkit (NLTK) is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources (like WordNet), along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning.

Key Features:

  • Tokenization: Splitting text into words or sentences.
  • Stemming and Lemmatization: Reducing words to their root form.
  • Part-of-Speech Tagging: Identifying the grammatical role of words.
  • Named Entity Recognition (NER): Identifying and classifying named entities in text.
  • Parsing: Analyzing the grammatical structure of sentences.
  • Classification: Building models to categorize text.
  • Semantic Reasoning: Tools for understanding word meanings and relationships (e.g., WordNet).
  • Corpus Access: Easy access to a wide variety of linguistic corpora.

Getting Started: Installation

You can install NLTK using pip. After installation, you'll typically need to download some NLTK data (corpora, models, etc.).

Using pip:

pip install nltk

Downloading NLTK Data (Python interactive shell):

import nltk
nltk.download('punkt')       # For tokenization
nltk.download('wordnet')     # For lemmatization and semantic reasoning
nltk.download('averaged_perceptron_tagger') # For POS tagging
nltk.download('stopwords')   # For common words filtering
nltk.download('maxent_ne_chunker') # For Named Entity Recognition
nltk.download('words')       # Required for some NER operations

Basic Concepts: Tokenization and Stemming

Tokenization

Tokenization is the process of breaking a stream of text into words, phrases, symbols, or other meaningful elements called tokens.

import nltk
# Ensure 'punkt' is downloaded: nltk.download('punkt')

from nltk.tokenize import word_tokenize, sent_tokenize

text = "Hello, world! NLTK is a powerful library. It helps with NLP tasks."

# Word tokenization
words = word_tokenize(text)
print("Words:", words)

# Sentence tokenization
sentences = sent_tokenize(text)
print("Sentences:", sentences)

Stemming

Stemming is the process of reducing inflected (or sometimes derived) words to their word stem, base or root form.

import nltk
from nltk.stem import PorterStemmer, LancasterStemmer

# Porter Stemmer (most common)
porter = PorterStemmer()
stemmed_words_porter = [porter.stem(w) for w in ["program", "programming", "programs", "programmer"]]
print("Porter Stemmer:", stemmed_words_porter)

# Lancaster Stemmer (more aggressive)
lancaster = LancasterStemmer()
stemmed_words_lancaster = [lancaster.stem(w) for w in ["program", "programming", "programs", "programmer"]]
print("Lancaster Stemmer:", stemmed_words_lancaster)

Lemmatization

Lemmatization is the process of grouping together the inflected forms of a word so they can be analyzed as a single item, identified by the word's lemma, or dictionary form. Unlike stemming, lemmatization considers the context and converts the word to its meaningful base form.

import nltk
# Ensure 'wordnet' is downloaded: nltk.download('wordnet')

from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()

print("Lemmatizing 'running':", lemmatizer.lemmatize("running", pos="v")) # v for verb
print("Lemmatizing 'better':", lemmatizer.lemmatize("better", pos="a")) # a for adjective
print("Lemmatizing 'cars':", lemmatizer.lemmatize("cars")) # default is noun

Further Topics:

  • Part-of-Speech Tagging
  • Named Entity Recognition (NER)
  • Chunking and Chinking
  • Parsing (Constituency and Dependency)
  • WordNet for Semantic Analysis
  • Text Classification
  • Sentiment Analysis
  • Working with NLTK Corpora

This document provides a basic introduction to NLTK. More detailed topics, advanced techniques, and practical examples will be covered in subsequent files.