Everything in your two units, in the order the syllabus lists it, with the three interactive pieces that are genuinely hard to picture from a textbook — activations and their gradients, the convolution window, and why momentum beats plain SGD.
A deep neural network is a chain of alternating linear maps and non-linear squashes. One layer is:
with a[0] = x, the input. Stack L of these and the final a[L] is the prediction ŷ. That is the entire forward pass.
Without the non-linear g, stacking layers is pointless. Two linear layers compose into W₂(W₁x + b₁) + b₂, which is just another linear map W′x + b′. A hundred linear layers still only draws a straight line. The activation function is what makes depth mean anything — every layer you add can then bend the decision surface again.
Two terms that get confused in vivas:
W and b. Learned by gradient descent.Parameter count for a dense layer is (inputs × units) + units. A 784→128 layer is 784×128 + 128 = 100,480 parameters — useful to be able to compute on the spot, because examiners ask.
Pick a function below. The solid line is g(x); the dashed line is its derivative g′(x) — and the derivative is the part that matters, because it is what gets multiplied down the chain during backpropagation.
Watch the dashed line. Wherever it sits near zero, gradient stops flowing — that region is where learning dies.
| Function | Range | Use it for | Its failure mode |
|---|---|---|---|
Sigmoid1/(1+e⁻ˣ) | (0, 1) | Binary classification output only | Saturates at both ends; max derivative is 0.25, so gradients shrink by ≥4× per layer. Not zero-centred. |
Tanh(eˣ−e⁻ˣ)/(eˣ+e⁻ˣ) | (−1, 1) | Hidden layers in RNNs; still standard inside LSTM/GRU | Also saturates, but is zero-centred, so it converges faster than sigmoid. |
ReLUmax(0, x) | [0, ∞) | Default for hidden layers in CNNs and MLPs | Dying ReLU — a unit stuck at negative input has derivative exactly 0 forever. |
Leaky ReLUmax(0.01x, x) | (−∞, ∞) | When you suspect dead units | The slope 0.01 is one more hyperparameter to guess. |
ELUx if x>0 else α(eˣ−1) | (−α, ∞) | Smoother alternative to ReLU; pushes mean activation toward zero | The exponential costs more compute than ReLU. |
Softmaxeᶻᵢ/Σeᶻʲ | (0,1), sums to 1 | Multi-class output layer only | Never use it in a hidden layer — it couples all units together. |
Backprop multiplies derivatives layer by layer. With sigmoid, each factor is at most 0.25. Across 10 layers the gradient reaching layer 1 is scaled by at most 0.2510 ≈ 10−6 — the early layers effectively stop learning. ReLU fixes this because its derivative is exactly 1 for positive inputs, so the product does not decay. This single fact is why ReLU replaced sigmoid in hidden layers, and it is the most commonly asked "why" in this unit.
Your MNIST/CIFAR configurations use activation="relu" throughout except two that switch to "elu". That is the comparison this section explains: ELU allows small negative outputs, which keeps the mean activation nearer zero and can train slightly faster on deeper stacks — at the cost of an exponential per unit. Your deeper 4- and 6-layer configs are exactly where a difference would show, if one shows at all.
The loss measures how wrong one prediction is. The cost is the loss averaged over the batch. Gradient descent minimises the cost.
Huber is the compromise: quadratic near zero so it has a smooth, well-behaved gradient, linear far away so a single freak outlier cannot dominate the update. That is precisely why it is the standard choice for noisy real-world series.
Two reasons, and examiners want both. (1) Gradient quality: pair sigmoid/softmax with MSE and the derivative contains a g′(z) factor that vanishes exactly when the prediction is confidently wrong — the network learns slowest when it most needs to learn. Cross-entropy is constructed so that factor cancels, leaving the clean gradient (ŷ − y). (2) Principle: cross-entropy is the negative log-likelihood of the correct class, so minimising it is maximum-likelihood estimation. MSE assumes Gaussian noise, which is the wrong model for a categorical label.
Your configs switch between "categorical_crossentropy" and "sparse_categorical_crossentropy". They compute the same quantity. The only difference is the label format: categorical needs one-hot labels ([0,0,1,0]), sparse takes the plain integer (2). Sparse saves memory on CIFAR-100. Getting this pairing wrong is the single most common silent bug in a Keras classifier — it will either crash on shape or train to nonsense.
Training is one loop, repeated:
ŷ.∂L/∂W for every weight.Step 3 is backpropagation, and it is nothing but the chain rule applied efficiently. For a weight in layer l:
The clever part is that the shared prefix of that product is reused for every layer, so the whole gradient costs about as much as one extra forward pass rather than one pass per weight. Define the error signal δ[l] = ∂L/∂z[l] and it becomes a clean two-line recursion:
Then the update, with learning rate η:
Notice g′(z) appearing in every backward step. That is the same factor from section 2 — the link between "which activation did I pick" and "does this network train at all" runs straight through this recursion.
Regression predicts a continuous number. Three design rules follow from that and they are not negotiable:
from tensorflow.keras import layers, models
model = models.Sequential([
layers.Input(shape=(n_features,)),
layers.Dense(64, activation="relu"),
layers.Dense(64, activation="relu"),
layers.Dense(1) # linear: no activation
])
model.compile(optimizer="adam", loss="mse", metrics=["mae"])Inputs are a dozen numeric features: rooms, age, distance to employment centres, median income. Target is a price in lakhs or thousands of dollars. Standardise every input to zero mean and unit variance first. Report MAE rather than MSE when you present it, because MAE is in the same unit as the target and therefore means something to a reader — "off by ₹2.3 lakh on average" is interpretable; "MSE 5.29" is not.
Fit the scaler on the training set only, then apply it to validation and test. Fitting it on everything leaks information about the test distribution into training, and your reported error becomes optimistic and wrong. This gets marked in lab vivas.
Classification predicts a class label. The design rules mirror regression exactly:
| Problem | Output layer | Loss |
|---|---|---|
| Binary (2 classes) | 1 unit, sigmoid | binary_crossentropy |
| Multi-class, one label | k units, softmax | categorical_crossentropy (one-hot) or sparse_categorical_crossentropy (integer) |
| Multi-label, several at once | k units, sigmoid | binary_crossentropy |
That last row is the one people get wrong. Softmax forces the outputs to sum to 1, so it cannot express "this image contains both a cat and a dog". Independent sigmoids can.
model = models.Sequential([
layers.Input(shape=(784,)),
layers.Dense(128, activation="relu"),
layers.Dropout(0.3),
layers.Dense(10, activation="softmax") # k units, softmax
])
model.compile(optimizer="adam",
loss="sparse_categorical_crossentropy",
metrics=["accuracy"])Flatten each 28×28 image to a 784-vector, two hidden layers, softmax over 10 classes. This reaches roughly 97–98% test accuracy. Worth knowing as a baseline, because Unit III's whole argument is that a CNN beats it while using fewer parameters — and the reason is that flattening throws away the fact that neighbouring pixels are related.
On a balanced set like MNIST, accuracy is fine. On an imbalanced one — say 2% of samples are the disease-positive class — a model predicting "negative" every single time scores 98% accuracy and is useless. Then you report precision, recall and F1, and you look at the confusion matrix. For a bioinformatics dataset this is nearly always the case, so expect it as a viva follow-up.
Plain gradient descent uses the whole dataset for one step — accurate but far too slow. The variants:
| Variant | Batch used per update | Character |
|---|---|---|
| Batch GD | All n samples | Smooth, slow, memory-hungry |
| Stochastic GD | 1 sample | Very noisy, but the noise can escape shallow local minima |
| Mini-batch SGD | 32–256 samples | The practical default — what "SGD" means in every framework |
Plain SGD oscillates across a narrow valley instead of running along it. Momentum accumulates a velocity so that consistent directions build up and oscillating ones cancel:
The physical reading is a ball rolling downhill with inertia rather than a hiker recomputing the steepest direction at every footstep. Nesterov momentum refines it by evaluating the gradient at the point the velocity is about to carry you to — a look-ahead correction.
One global η is wrong when different parameters need different step sizes. These give every parameter its own:
The valley is 12× steeper across than along — the classic hard case. All three start at the same point with the same learning rate. Watch how plain SGD zig-zags while momentum straightens out.
Eleven of your twelve RNN configs use Adam at 1e-3; config 12 switches to SGD(learning_rate=1e-5, momentum=0.9). That is not a typo — it is the point of the comparison. Adam's per-parameter scaling means its η is not comparable to SGD's, and recurrent networks have notoriously badly scaled gradients, so a raw SGD step that works at all has to be roughly a hundred times smaller. Running it is how you demonstrate that adaptivity is doing real work, not just cosmetic tuning.
Feed a 224×224 colour image to a dense layer of 1000 units and you need 224×224×3×1000 ≈ 150 million weights in the first layer alone. It will not fit, will not train, and will not generalise. Convolution fixes this with three ideas:
A 3×3 kernel over 3 channels producing 32 feature maps costs 3×3×3×32 + 32 = 896 parameters — regardless of whether the image is 32×32 or 4000×4000.
Slide a small kernel over the input; at each position compute the element-wise product and sum it. Click cells in the input below to redraw the shape, and change the kernel to see what each one detects.
Click any input cell to toggle it. The highlighted 3×3 window shows which input patch produced the highlighted output cell — hover the output to move it.
"same" padding means choosing P so the output matches the input size — for stride 1 that is P = (K−1)/2, so 1 for a 3×3 kernel and 2 for a 5×5. "valid" means no padding, and the map shrinks by K−1 each layer.
Two stacked 3×3 layers see the same 5×5 region as one 5×5 layer, but use 2×(3×3)=18 weights per channel pair instead of 25, and apply a non-linearity in between — more expressive, fewer parameters. Three stacked 3×3s match a 7×7 at 27 weights versus 49. This is why modern architectures are almost entirely 3×3, and why your kernels=[7,7] and [7,7,7] configs are expected to underperform the deep 3×3 stacks at similar cost. Being able to state that trade-off is a guaranteed exam answer.
Pooling downsamples a feature map by summarising each small window with a single number.
A 2×2 pool with stride 2 halves both spatial dimensions and so quarters the number of activations. It has no learnable parameters — a frequent one-mark question.
What pooling buys you: a wider receptive field for later layers, less computation, and a degree of translation invariance — shift the input by one pixel and a max-pooled output often does not change at all.
These are examined as a pair and they are not the same. Convolution is equivariant: shift the input, the output shifts identically. Pooling adds invariance: shift the input slightly, the output does not change. Convolution tracks where; pooling lets the network stop caring exactly where.
The standard pattern, repeated in blocks:
Filters double as spatial size halves — 32, 64, 128, 256 — keeping roughly constant computation per block while the features become more abstract: edges, then textures, then parts, then objects.
model = models.Sequential([
layers.Input(shape=(32, 32, 3)),
layers.Conv2D(32, (3,3), padding="same", use_bias=False),
layers.BatchNormalization(),
layers.Activation("relu"),
layers.MaxPooling2D((2,2)), # 32x32 -> 16x16
layers.Dropout(0.2),
layers.Conv2D(64, (3,3), padding="same", use_bias=False),
layers.BatchNormalization(),
layers.Activation("relu"),
layers.MaxPooling2D((2,2)), # 16x16 -> 8x8
layers.Dropout(0.3),
layers.Flatten(),
layers.Dense(128), layers.BatchNormalization(), layers.Activation("relu"),
layers.Dropout(0.4),
layers.Dense(10, activation="softmax"),
])use_bias=False on the conv beneath it — BN has its own shift parameter, so the conv bias is redundant. Your notebook does exactly this.Your grid varies depth, kernel size, filter count, dropout, batch norm and activation. When you write it up, the defensible claims are comparisons between neighbouring rows that differ in one thing — that is how the grid was built. Expect: MNIST saturates around 99%+ so differences there are noise; CIFAR-10 is where depth and batch norm separate the configurations; CIFAR-100 is where every configuration struggles, because 100 classes over the same 50,000 images means only 500 examples per class. If your no-dropout, no-augmentation configs show training accuracy far above validation accuracy, that gap is the overfitting result — report it rather than hiding it.
This syllabus line asks a real question: must convolution kernels be learned by supervised backpropagation?
Labelled biological images are scarce and expensive — someone has to annotate them. Unlabelled ones are abundant. Unsupervised and self-supervised pretraining is how the field gets around that, and it is a strong point to make if this comes up as a discussion question.
A CNN sees a fixed-size input all at once. Sequences — text, speech, a time series — have variable length and order that carries meaning. An RNN handles this by keeping a hidden state that it carries forward:
hₜ is a summary of everything seen up to time t. Crucially the same W matrices are reused at every timestep — parameter sharing across time, exactly as convolution shares across space.
Training unrolls the loop into a deep feedforward network, one layer per timestep, and backpropagates through it. That is backpropagation through time (BPTT). And it inherits a serious problem: the gradient across k steps contains Wₕₕ raised to the power k.
If the dominant eigenvalue of Wₕₕ is below 1, the gradient vanishes exponentially and the network cannot learn dependencies more than roughly 10 steps back. If it is above 1, the gradient explodes into NaNs. Exploding gradients have a cheap fix — gradient clipping, rescale any gradient whose norm exceeds a threshold. Vanishing gradients do not; they need an architectural change, and that change is the LSTM.
The LSTM adds a second state: a cell state Cₜ that runs straight through the unit with only element-wise multiplication and addition on its path. No repeated matrix multiplication means no exponential decay — gradients can flow across hundreds of steps. Three gates, each a sigmoid producing values in (0,1), decide what passes.
"The cell state is modified only by element-wise multiplication by the forget gate and element-wise addition of the gated candidate. Because there is no repeated matrix multiplication along that path, the gradient does not decay exponentially, so the LSTM can learn dependencies over hundreds of timesteps where a simple RNN manages about ten."
One practical note: initialise the forget-gate bias to 1 so the cell remembers by default at the start of training. Keras does this for you via unit_forget_bias=True.
The GRU is a simplification of the LSTM. Two changes: there is no separate cell state — hₜ does both jobs — and the forget and input gates are merged into a single update gate, since deciding how much to keep already decides how much to admit.
| LSTM | GRU | |
|---|---|---|
| Gates | 3 (forget, input, output) | 2 (update, reset) |
| States | 2 (cell C and hidden h) | 1 (h only) |
| Parameters | 4 weight blocks | 3 blocks — about 25% fewer |
| Speed | Slower | Faster to train |
| When to prefer | Long sequences, plenty of data | Smaller datasets, tighter compute |
Empirically they perform comparably on most tasks, and there is no reliable rule for which wins — which is exactly why configs 1, 2 and 3 of your sunspots grid swap SimpleRNN → LSTM → GRU with everything else held fixed. That three-row comparison is the experiment; the honest write-up says which won on your data and does not over-generalise from it.
An RNN layer either returns its output at every timestep or only the last one. This is the single detail that breaks most student implementations:
return_sequences | Output shape | Use when |
|---|---|---|
False (default) | (batch, units) | It is the last recurrent layer, feeding a Dense head |
True | (batch, timesteps, units) | Another recurrent layer follows, or you need an output per step |
Stacking rule: every recurrent layer except the last needs return_sequences=True.
A forecasting model needs (X, y) pairs, so you slide a window across the series: the previous w values are the input, the next value is the target. Window 30 on monthly data means "predict from the last 30 months".
def windowed_dataset(series, window_size, batch_size):
ds = tf.data.Dataset.from_tensor_slices(series)
ds = ds.window(window_size + 1, shift=1, drop_remainder=True)
ds = ds.flat_map(lambda w: w.batch(window_size + 1))
ds = ds.map(lambda w: (w[:-1], w[-1])) # inputs, label
return ds.batch(batch_size).prefetch(1)Split chronologically — train on the earliest portion, validate on the most recent. A random split lets the model see the future while training, which is data leakage; your validation error will look excellent and the model will be worthless. Shuffling within the already-split training windows is fine and helps optimisation; shuffling before the split is the error.
Train on months 0–3000, validate on the last 235 — correctly chronological. Then the interesting choices:
mae alongside the loss, because MAE is in sunspot units and is what you quote in a conclusion.A standard RNN reads left to right, so at time t it knows only the past. A bidirectional RNN runs two independent recurrent layers — one forward, one backward over the reversed sequence — and concatenates their hidden states:
layers.Bidirectional(layers.LSTM(64, return_sequences=True))Every output now depends on the entire sequence, past and future. That resolves ambiguities a forward-only model cannot: in "the bank of the river", the disambiguating word arrives afterwards.
A bidirectional model needs the whole sequence before it can produce any output. So it is excellent for offline, complete sequences — named-entity recognition, sentiment classification, protein secondary-structure prediction from a full sequence, sentence translation. It is unusable for real-time forecasting, because the "future" it wants to read is the thing you are trying to predict. For genuine live prediction it is cheating; for a retrospective study it is not.
This is worth being careful about in your own sunspots work: config 9 is bidirectional, and it may well score best on the leaderboard. If it does, the correct write-up says it is not a valid forecaster despite the number, because within each 60-month window it is reading months that would not have happened yet at prediction time. Saying that yourself is a stronger result than the score.
In bioinformatics the offline case is the common one: a BiLSTM over a protein sequence is standard for secondary-structure and binding-site prediction, precisely because the whole sequence is already known.
O = floor((W − K + 2P)/S) + 1(K × K × C_in × C_out) + C_out(inputs × units) + unitshₜ = (1−z)⊙hₜ₋₁ + z⊙h̃ₜm, v, bias correction, and updateWork through section 9's output-size calculator until the arithmetic is automatic, then write out the LSTM equations from memory three times. Those two things between them cover more marks than anything else in these two units.