AI How Neural Networks Learn

The Forward Pass: Making a Guess

Every time data flows through a network from the input layer to the output layer, that is called a forward pass. Each layer takes the previous layer's numbers, applies its weights and activation function, and passes the result onward, until the output layer produces a prediction. At the very start of training, the weights are set to small random values, so this first prediction is essentially a guess - usually a bad one.

Measuring the Error With a Loss Function

To know how bad the guess was, the network compares its prediction to the correct answer using a loss function (sometimes called a cost function). The loss function outputs a single number: the bigger the number, the further the prediction was from the truth. Different tasks use different loss functions.

Backpropagation: Working Out Who's to Blame

Once the loss is known, the network needs to figure out how much each individual weight contributed to that error, so it knows which weights to adjust and by how much. Backpropagation is the algorithm that does this: it starts at the output layer and works backward through the network, layer by layer, calculating each weight's share of the blame for the total error. Under the hood it relies on the chain rule from calculus to pass that information backward efficiently - you don't need to work through that math by hand, every deep learning framework does it automatically, but it's worth knowing that backpropagation is really just an efficient bookkeeping method for answering: 'if this one weight changed slightly, how much would the final error change?' for every weight in the network at once.

Gradient Descent: Small Steps Downhill

Backpropagation tells the network the direction each weight should move to reduce the error; gradient descent is the rule that actually moves it. Picture the loss function as a hilly landscape, where height represents how wrong the model currently is. Gradient descent takes a small step for every weight in the downhill direction - the direction that reduces the error the fastest - then the whole forward pass, loss calculation, and backward pass repeat again. Over many repetitions the network gradually rolls toward a low point in that landscape, where its predictions are much closer to correct.

# A simplified illustration of gradient descent on ONE weight.
# Real networks repeat a version of this for thousands (or billions) of
# weights at once, using backpropagation to work out each weight's share
# of the error.

weight = 0.2
learning_rate = 0.01
input_value = 5
target = 10  # we want weight * input_value to reach this

for step in range(5):
    prediction = weight * input_value
    error = target - prediction
    gradient = -2 * input_value * error   # slope of the error curve here
    weight = weight - learning_rate * gradient
    print(f"Step {step}: weight={weight:.3f}, prediction={prediction:.2f}, error={error:.2f}")

• Epoch - one complete pass of the whole training dataset through the network
• Batch - a small group of examples processed together before the weights are updated once
• Learning rate - a small number that controls how big each weight-update step is
• Convergence - the point where further training barely reduces the loss any further