How to use the Activation Function Plotter
- 1. Tick functionsChoose which activation functions to compare.
- 2. Set the rangePick the x range and whether to show functions, derivatives or both.
- 3. Plot and explorePlot once, then change ticks and values to update live.
Formulas
- ReLU: max(0, x). Derivative 1 for x > 0, else 0.
- Sigmoid: 1 ÷ (1 + e⁻ˣ), output 0 to 1. Derivative σ(x)(1 − σ(x)), at most 0.25.
- Tanh: (eˣ − e⁻ˣ) ÷ (eˣ + e⁻ˣ), output −1 to 1.
- GELU: x × Φ(x), where Φ is the normal CDF. Used in BERT and GPT-style models.
- Swish / SiLU: x × σ(x). Softplus: ln(1 + eˣ), a smooth ReLU.
At x = 1, sigmoid gives 0.7311, tanh gives 0.7616, ReLU gives 1, GELU about 0.8413 and softplus 1.3133. The table under the chart shows these for whatever x you choose.
Why the choice matters
Switch to derivative view and look at sigmoid and tanh: their gradients fall towards zero for large |x|. In a deep network, multiplying many small gradients together makes early layers learn very slowly; this is the vanishing gradient problem. ReLU keeps a gradient of exactly 1 for positive inputs, which is a big reason deep networks became trainable.
ReLU has its own weakness: neurons stuck at negative inputs output 0 and get zero gradient forever (dying ReLU). Leaky ReLU, ELU, GELU and swish keep a small slope or a smooth curve below zero. Sigmoid is still used on the output of binary classifiers, and softmax on multi-class outputs; try the softmax calculator. See activations at work in the neural network playground.
Frequently asked questions
Which activation should I use in hidden layers?
ReLU is the usual default for CNNs and MLPs. GELU is common in transformers. Tanh suits small networks and some RNNs.
Is GELU exact here?
It uses x × Φ(x) with a precise error-function approximation, accurate to about seven decimal places.
Is the Activation Function Plotter free?
Yes, the Activation Function Plotter is free. The first time you plot your first chart, you are asked for your name and WhatsApp number once. After that, this and every other Uncodemy tool opens straight away without the form.