Neural networks explained

A neural network is a function built from many small, identical pieces. Understanding one piece — the neuron — and how pieces are stacked and trained demystifies everything from a simple classifier to GPT.

One neuron

A neuron takes inputs, multiplies each by a weight, adds a bias and passes the result through an activation function:

z = w1*x1 + w2*x2 + ... + wn*xn + b
output = activation(z)

Imagine predicting whether a customer will buy a festive gift box. Inputs could be past spend, days to Diwali and whether they opened the last SMS. The weights express how much each input matters; the bias shifts the decision threshold.

Layers and depth

Neurons are grouped into layers. A fully connected (dense, nn.Linear) layer connects every input to every neuron. Stacking layers gives:

• Input layer – the raw features or pixels.
• Hidden layers – learn intermediate representations.
• Output layer – one neuron for regression, one per class for classification.

Without an activation between layers, stacking is pointless: two linear layers collapse into one linear layer. Activations add the non-linearity that lets networks model curves, interactions and complex boundaries.

Activation functions

Activation	Formula (informal)	Range	Typical use
ReLU	max(0, z)	0 to infinity	Default for hidden layers
Leaky ReLU / GELU	small slope for negatives / smooth ReLU	roughly -inf to inf	Transformers use GELU
Sigmoid	1 / (1 + e^-z)	0 to 1	Binary output probability
Tanh	(e^z - e^-z) / (e^z + e^-z)	-1 to 1	Inside RNN/LSTM cells
Softmax	e^zi / sum of e^zj	0 to 1, sums to 1	Multi-class output

The forward pass

Data flows from input to output, producing a prediction. A loss function then scores how wrong it is — mean squared error for regression, cross-entropy for classification.

Backpropagation: the intuition

We want to know, for every weight, "if I increase this weight slightly, does the loss go up or down, and by how much?" That quantity is the partial derivative of the loss with respect to the weight — its gradient.

Backpropagation computes all gradients efficiently using the chain rule. Picture a chain: weight w affects z, z affects the activation a, a affects the prediction, and the prediction affects the loss. The gradient of the loss with respect to w is the product of the local derivatives along that chain:

dLoss/dw = dLoss/dpred * dpred/da * da/dz * dz/dw

Starting from the loss, the algorithm walks backwards layer by layer, reusing already computed products, so the cost is only about twice a forward pass — even for millions of weights.

Gradient descent

With gradients in hand, each weight takes a small step downhill:

w_new = w_old - learning_rate * gradient

Think of standing on a foggy hill in Munnar and feeling the slope under your feet: you step in the steepest downward direction, repeat, and eventually reach a valley. The learning rate is your stride. Too large and you overshoot; too small and training crawls.

A tiny network in code

import torch
import torch.nn as nn

class TinyNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden = nn.Linear(3, 8)
        self.out = nn.Linear(8, 1)

    def forward(self, x):
        h = torch.relu(self.hidden(x))
        return torch.sigmoid(self.out(h))

net = TinyNet()
x = torch.tensor([[2.5, 12.0, 1.0]])     # spend (k INR), days to Diwali, opened SMS
print(net(x))                             # untrained probability
print(sum(p.numel() for p in net.parameters()))   # 41 parameters

The count is 3x8 + 8 weights and biases in the hidden layer plus 8x1 + 1 in the output layer = 41.

Why depth helps — and hurts

Deeper networks can represent more complex functions with fewer total neurons, because later layers reuse features built by earlier ones. But very deep plain networks suffer from vanishing gradients: multiplying many small derivatives (for example, sigmoid's maximum slope is 0.25) makes early layers learn almost nothing. ReLU, careful weight initialisation, batch normalisation and residual (skip) connections are the standard cures.

Key takeaways

• A neuron is a weighted sum plus bias, passed through a non-linear activation.
• Layers stack neurons; non-linearity is what makes depth meaningful.
• Backpropagation applies the chain rule backwards to get every gradient cheaply.
• Gradient descent uses those gradients and a learning rate to update weights.