TF-IDF

TF-IDF (Term Frequency-Inverse Document Frequency) is a weighting scheme that scores how important a word is to a specific document within a larger corpus - giving higher weight to words that appear often in one document but rarely across the rest of the corpus.

The Two Components

Term Frequency (TF) - how often a term appears in a given document
Inverse Document Frequency (IDF) - a measure that reduces the weight of terms that appear in many documents (like common words) and increases the weight of rarer, more distinctive terms

The Formula

TF-IDF(t, d) = TF(t, d) x log(N / DF(t))

# t  = term
# d  = document
# N  = total number of documents
# DF(t) = number of documents containing term t

Computing TF-IDF in Python

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd

docs = ["data science is fun", "python for data science"]
vectorizer = TfidfVectorizer()
matrix = vectorizer.fit_transform(docs)

tfidf_df = pd.DataFrame(matrix.toarray(), columns=vectorizer.get_feature_names_out())
print(tfidf_df)

Why TF-IDF Beats Plain Word Counts

Unlike a simple Bag of Words count, TF-IDF automatically down-weights common words (like "data" if it appears in every document) and highlights terms that are genuinely distinctive to a particular document - making it especially useful for search ranking, keyword extraction, and text classification.

You've Completed This Section

This wraps up the Text Mining and NLP foundations - from understanding text data sources and the Bag of Words model, through core NLP terminologies, to representing text numerically with DTM, TDM, and TF-IDF. Together, these techniques form the essential toolkit for turning unstructured text into data a machine learning model can learn from.