Information Gain

Information Gain measures how much a split reduces entropy - it's the metric a Decision Tree uses to decide which feature and threshold to split on at each node.

The Information Gain Formula

InformationGain = Entropy(parent) - WeightedAverageEntropy(children)

# A higher information gain means the split does a better job
# of separating the classes.

Calculating Information Gain in Python

import numpy as np

def entropy(labels):
    values, counts = np.unique(labels, return_counts=True)
    probs = counts / counts.sum()
    return -np.sum(probs * np.log2(probs))

def information_gain(parent, left, right):
    weight_left = len(left) / len(parent)
    weight_right = len(right) / len(parent)
    return entropy(parent) - (weight_left * entropy(left) + weight_right * entropy(right))

parent = [0, 0, 0, 1, 1, 1, 1, 1]
left   = [0, 0, 0]
right  = [1, 1, 1, 1, 1]

print("Information Gain:", information_gain(parent, left, right))

How the Tree Uses It

At each node, the Decision Tree evaluates information gain for every possible feature and split point, then picks whichever split produces the highest gain.

Coming Up Next

Next, you'll learn about the Greedy Approach that Decision Trees follow when choosing splits.