Entropy

Entropy is a measure of impurity or randomness in a dataset, and it's the concept Decision Trees use to decide how "mixed" a node's classes are before and after a split.

The Entropy Formula

Entropy(S) = -sum(p_i * log2(p_i))

# S    = the dataset (or node) being measured
# p_i  = proportion of class i in S

Interpreting Entropy Values

Entropy = 0     -> node is pure (all samples belong to one class)
Entropy = 1     -> maximum impurity for a binary split (classes are 50/50)

Calculating Entropy in Python

import numpy as np

def entropy(labels):
    values, counts = np.unique(labels, return_counts=True)
    probs = counts / counts.sum()
    return -np.sum(probs * np.log2(probs))

labels = [0, 0, 0, 1, 1, 1, 1, 1]
print("Entropy:", entropy(labels))

Why Entropy Matters

A Decision Tree tries to reduce entropy at every split - the goal is to move from a mixed, uncertain node toward child nodes that are as pure as possible.

Entropy is just one way to measure impurity - Gini impurity is another common alternative that's computationally slightly cheaper and often gives similar results.

Coming Up Next