Lesson 2 — How Models Learn Patterns
Learning Outcome: Explain prediction → loss → gradient descent → backpropagation → regularization; build intuition for how structure emerges during training.
Manual
The Law of Adjustment
All intelligences learn by making predictions, comparing them to reality, and adjusting themselves to reduce error. This loop is the mechanical mirror of growth: expectation, feedback, correction.
1 · The Loop of Prediction
A model guesses a label/word/value, then measures the gap with a loss function. Loss isn't failure; it's a numerical signal of what remains to be learned.
- ML: prediction, loss, objective
- Cognitive: perception notices mismatch; attention shifts
- Philosophy: error as information
2 · Gradient Descent
A gradient is the slope of loss with respect to parameters. Gradient descent takes small steps downhill. This is learning as guided drift: motion toward lower confusion.
- ML: gradient, learning rate, convergence
- Philosophy: becoming through iteration
3 · Backpropagation
Backpropagation distributes responsibility for error backward through the network. Each weight updates in proportion to its contribution—synthetic reflection.
- ML: chain rule, partial derivatives, layers
- Quantum: measurement analog—local collapses that shape the global state (metaphor)
4 · Regularization
Unconstrained models memorize noise (overfitting). Regularization rewards simplicity and penalizes excess capacity. It prefers essence over appearance—biasing the system toward general truths.
- ML: L2/L1, dropout, early stopping
- Cognitive: abstraction discards detail to keep meaning
5 · Emergent Representation
As loss drops, hidden layers stabilize into representations—recurring patterns encoded as geometry. Structure wasn't programmed; it emerged from experience.
Workshop
Hands-On: See Learning Happen (Loss vs Epoch)
Goal: build intuition for loss curves and overfitting using a tiny synthetic task—no server training required.
Option A — In-browser demo
Renders a toy regression with SGD on noisy points and plots loss per epoch.
Loss function visualization placeholder
(Full interactive demo available at brainfoundry.ai)
Observe
- Early epochs: steep loss drop (learning signal)
- Mid epochs: diminishing returns (fine-tuning)
- Push too far or lower noise → watch overfit appear (loss ↓ on train, ↑ on hold-out)
Option B — Notebook (local)
Run the provided minimal notebook to:
- Fit y = 2x + ε with a 1-hidden-layer net
- Plot train vs. validation loss
- Toggle L2 and early stopping to see smoothing
Why no Foundry endpoint here?
BrainFoundry focuses on RAG, embeddings, search, deploy. Training here is didactic; we simulate locally to illustrate the principles without promising hosted training.
Reflection
Reflection — Your Learning Curve
Where in your process do you overfit to the past—and where do you regularize away useful novelty? Write 6–10 sentences linking one real habit to loss, descent, and regularization. Name one adjustment you'll try this week.