Backpropagation: A Complete Step-by-Step Manual Calculation

Backpropagation is the engine behind every modern neural network. Yet most tutorials skip it with abstract matrix notation. In this article we will compute every single step — digit by digit — so nothing is left as a black box.

Network Setup

We will use a simple network: 2 inputs → 2 hidden neurons → 1 output, with Sigmoid activation at every neuron.

Initial Values

Sigmoid Activation Function

σ(x) = 1 / (1 + e^(−x))

Its derivative is elegantly expressed in terms of the output itself:

σ'(x) = σ(x) · (1 − σ(x))

Part 1 — Forward Pass

The forward pass computes the prediction by flowing data from inputs to output.

Step 1.1 — Hidden Neuron h₁

Compute the net input (weighted sum):

net_h1 = (w₁ · x₁) + (w₂ · x₂) + b_h
       = (0.15 × 0.05) + (0.20 × 0.10) + 0.35
       = 0.0075 + 0.0200 + 0.3500
       = 0.3775

Apply the Sigmoid activation:

out_h1 = σ(0.3775)
       = 1 / (1 + e^(−0.3775))
       = 1 / (1 + 0.68566)
       = 1 / 1.68566
       = 0.59327

Step 1.2 — Hidden Neuron h₂

net_h2 = (w₃ · x₁) + (w₄ · x₂) + b_h
       = (0.25 × 0.05) + (0.30 × 0.10) + 0.35
       = 0.0125 + 0.0300 + 0.3500
       = 0.3925

out_h2 = σ(0.3925)
       = 1 / (1 + e^(−0.3925))
       = 1 / (1 + 0.67530)
       = 1 / 1.67530
       = 0.59688

Step 1.3 — Output Neuron o₁

net_o1 = (w₅ · out_h1) + (w₆ · out_h2) + b_o
       = (0.40 × 0.59327) + (0.45 × 0.59688) + 0.60
       = 0.23731 + 0.26860 + 0.60000
       = 1.10591

out_o1 = σ(1.10591)
       = 1 / (1 + e^(−1.10591))
       = 1 / (1 + 0.33091)
       = 1 / 1.33091
       = 0.75137

Forward Pass Summary

Part 2 — Computing the Loss

We use Mean Squared Error (MSE):

E = 0.5 × (t₁ − out_o1)²
  = 0.5 × (0.01 − 0.75137)²
  = 0.5 × (−0.74137)²
  = 0.5 × 0.54963
  = 0.27482

The 0.5 factor is intentional — it cancels the exponent when we differentiate, giving a clean (t − out) instead of 2(t − out).

An error of 0.27482 means our prediction (0.75137) is far from the target (0.01). Backpropagation will correct every weight to minimize this error.

Part 3 — Backward Pass (Output Layer)

Goal: measure how much each weight contributed to the error, using the chain rule.

Step 3.1 — Gradient at the Output

We want ∂E/∂w₅. By the chain rule:

∂E/∂w₅ = ∂E/∂out_o1 × ∂out_o1/∂net_o1 × ∂net_o1/∂w₅

Term 1 — Derivative of error with respect to output:

∂E/∂out_o1 = −(t₁ − out_o1)
           = −(0.01 − 0.75137)
           = −(−0.74137)
           = +0.74137

Term 2 — Sigmoid derivative:

∂out_o1/∂net_o1 = out_o1 × (1 − out_o1)
                = 0.75137 × (1 − 0.75137)
                = 0.75137 × 0.24863
                = 0.18681

Term 3 — Derivative of net_o1 with respect to w₅:

∂net_o1/∂w₅ = out_h1 = 0.59327

Multiply all three terms:

∂E/∂w₅ = 0.74137 × 0.18681 × 0.59327
        = 0.13851 × 0.59327
        = 0.08217

Step 3.2 — Gradient for w₆

∂E/∂w₆ = ∂E/∂out_o1 × ∂out_o1/∂net_o1 × ∂net_o1/∂w₆
        = 0.74137 × 0.18681 × out_h2
        = 0.13851 × 0.59688
        = 0.08267

Note: the product ∂E/∂out_o1 × ∂out_o1/∂net_o1 = 0.13851 is called the output delta (δo) — we save it for use in the hidden layer.

Step 3.3 — Update Output Weights

w₅_new = w₅ − η × ∂E/∂w₅
       = 0.40 − (0.5 × 0.08217)
       = 0.40 − 0.04109
       = 0.35891

w₆_new = w₆ − η × ∂E/∂w₆
       = 0.45 − (0.5 × 0.08267)
       = 0.45 − 0.04134
       = 0.40866

Part 4 — Backward Pass (Hidden Layer)

Now we propagate the error back into the hidden layer. This is the trickier part — the error must travel backwards through the output weights.

Step 4.1 — Error Gradient for h₁

How much error flows back to h₁?

The error reaches h₁ only through w₅ (since there is a single output):

∂E/∂out_h1 = δo × w₅
           = 0.13851 × 0.40
           = 0.05540

Sigmoid derivative at h₁:

∂out_h1/∂net_h1 = out_h1 × (1 − out_h1)
                = 0.59327 × (1 − 0.59327)
                = 0.59327 × 0.40673
                = 0.24127

Hidden delta for h₁ (δh1):

δh1 = ∂E/∂out_h1 × ∂out_h1/∂net_h1
    = 0.05540 × 0.24127
    = 0.01337

Gradients for w₁ and w₂:

∂E/∂w₁ = δh1 × x₁ = 0.01337 × 0.05 = 0.000669
∂E/∂w₂ = δh1 × x₂ = 0.01337 × 0.10 = 0.001337

Step 4.2 — Error Gradient for h₂

∂E/∂out_h2 = δo × w₆
           = 0.13851 × 0.45
           = 0.06233

∂out_h2/∂net_h2 = out_h2 × (1 − out_h2)
                = 0.59688 × 0.40312
                = 0.24063

δh2 = 0.06233 × 0.24063
    = 0.01500

∂E/∂w₃ = δh2 × x₁ = 0.01500 × 0.05 = 0.000750
∂E/∂w₄ = δh2 × x₂ = 0.01500 × 0.10 = 0.001500

Step 4.3 — Update Hidden Weights

w₁_new = 0.15 − (0.5 × 0.000669) = 0.15 − 0.000335 = 0.149665
w₂_new = 0.20 − (0.5 × 0.001337) = 0.20 − 0.000669 = 0.199331
w₃_new = 0.25 − (0.5 × 0.000750) = 0.25 − 0.000375 = 0.249625
w₄_new = 0.30 − (0.5 × 0.001500) = 0.30 − 0.000750 = 0.299250

Part 5 — One Iteration Summary

Notice that w₅ and w₆ shift much more (±0.04) than w₁–w₄ (±0.0003). This makes sense — output layer weights have a direct influence on the error, while hidden layer weights are "diluted" by the intervening sigmoid.

Part 6 — Second Iteration (Convergence Check)

Use the updated weights for a second forward pass:

net_h1 = (0.149665 × 0.05) + (0.199331 × 0.10) + 0.35
       = 0.007483 + 0.019933 + 0.35
       = 0.377416

out_h1 = σ(0.377416) = 0.59315

net_h2 = (0.249625 × 0.05) + (0.299250 × 0.10) + 0.35
       = 0.012481 + 0.029925 + 0.35
       = 0.392406

out_h2 = σ(0.392406) = 0.59676

net_o1 = (0.35891 × 0.59315) + (0.40866 × 0.59676) + 0.60
       = 0.21291 + 0.24390 + 0.60
       = 1.05681

out_o1 = σ(1.05681) = 0.74217

Loss at iteration 2:

E₂ = 0.5 × (0.01 − 0.74217)²
   = 0.5 × (0.73217)²
   = 0.5 × 0.53607
   = 0.26804

Loss dropped from 0.27482 → 0.26804 ✓

After thousands of iterations the loss approaches zero and the output converges toward the target 0.01.

Part 7 — Full Backpropagation Flow

Part 8 — Python Verification

import numpy as np
 
def sigmoid(x):
    return 1 / (1 + np.exp(-x))
 
def sigmoid_deriv(x):
    s = sigmoid(x)
    return s * (1 - s)
 
# Initialization
x = np.array([0.05, 0.10])
t = 0.01
eta = 0.5
 
W_h = np.array([[0.15, 0.25],   # w1, w3 (columns = hidden neurons)
                [0.20, 0.30]])  # w2, w4
b_h = 0.35
 
W_o = np.array([0.40, 0.45])    # w5, w6
b_o = 0.60
 
print("=== Iteration 1 ===")
 
# Forward pass
net_h = x @ W_h + b_h           # [net_h1, net_h2]
out_h = sigmoid(net_h)           # [out_h1, out_h2]
 
net_o = out_h @ W_o + b_o        # net_o1
out_o = sigmoid(net_o)           # out_o1
 
print(f"out_h  : {out_h}")
print(f"out_o  : {out_o:.5f}")
 
# Loss
E = 0.5 * (t - out_o) ** 2
print(f"Loss   : {E:.5f}")
 
# Backward — output layer
delta_o = -(t - out_o) * sigmoid_deriv(net_o)
print(f"delta_o: {delta_o:.5f}")
 
grad_Wo = delta_o * out_h        # gradients for w5, w6
W_o_new = W_o - eta * grad_Wo
 
# Backward — hidden layer
delta_h = (delta_o * W_o) * sigmoid_deriv(net_h)
grad_Wh = np.outer(x, delta_h)  # shape (2, 2)
W_h_new = W_h - eta * grad_Wh
 
print(f"\nNew output weights : {W_o_new}")
print(f"New hidden weights :\n{W_h_new}")
 
# Verify iteration 2
net_h2 = x @ W_h_new + b_h
out_h2 = sigmoid(net_h2)
net_o2 = out_h2 @ W_o_new + b_o
out_o2 = sigmoid(net_o2)
E2 = 0.5 * (t - out_o2) ** 2
print(f"\n=== Iteration 2 ===")
print(f"out_o  : {out_o2:.5f}")
print(f"Loss   : {E2:.5f}")

Expected output:

=== Iteration 1 ===
out_h  : [0.59327 0.59688]
out_o  : 0.75137
Loss   : 0.27482
delta_o: 0.13851

New output weights : [0.35891 0.40866]
New hidden weights :
[[0.149665 0.249625]
 [0.199331 0.299250]]

=== Iteration 2 ===
out_o  : 0.74217
Loss   : 0.26804

The code output matches our manual calculation exactly. ✓

Why This Matters

GPT, YOLO, ResNet — all use the exact same mechanism. The only differences are:

• Number of layers (not 2, but 100+)
• Number of neurons (not 2, but millions)
• Activation functions (ReLU, GELU, SiLU — not Sigmoid)
• Optimizer (Adam, SGD — not vanilla gradient descent)

But the chain rule and the forward-backward flow are identical to what we just computed above.

Formula Reference