Lesson 3 — How They Reason and Generate
Learning Outcome: Understand how transformers use attention to model context, form reasoning chains, and generate coherent output.
Manual
The Architecture of Attention
Every act of reasoning begins as focus. An intelligence must decide where to look among its inputs. The transformer architecture encodes this decision mathematically through attention.
1 · From Sequence to Field
Earlier models processed tokens one after another. Transformers allow every token to attend to every other, turning sequence into relation. This shift mirrors awareness becoming contextual rather than linear.
2 · Self-Attention as Awareness
Each token computes queries, keys, and values, measuring relevance between them. Weights express importance; the weighted sum forms contextualized meaning.
- ML: attention weights, softmax normalization
- Cognitive: focus and selective memory
- Quantum: superposition of possibilities collapsed by weighting
3 · Multi-Head Perspective
Multiple heads attend from different angles—syntax, sentiment, rhythm. Their concatenation yields richer representation: distributed perception forming unity.
4 · Generation as Collapse
During decoding, probabilities across vocabulary represent words in superposition. Sampling collapses them—choice from potential. Temperature governs creativity; top-p limits chaos.
5 · Chain-of-Thought
Explicit reasoning emerges when intermediate steps are verbalized. Prompting a model to "think step by step" linearizes its internal web of attention. This externalizes the model's hidden structure, just as reasoning externalizes thought.
Workshop
Hands-On: Visualize Attention
Goal: inspect how a small transformer focuses across a sentence.
Option A — Browser Visualization
Render toy attention map for a short input text.
Attention mechanism visualization placeholder
(Full interactive demo available at brainfoundry.ai)
Observe
- Each row = a token attending to others
- Bright diagonals → local context; distant highlights → long-range reasoning
- Change text to see focus patterns shift
Option B — Foundry API
curl -X POST /api/foundry/analyze \
-H "Content-Type: application/json" \
-d '{ "text": "The forge refines itself through feedback." }'Returns normalized attention weights for visualization or further analysis.
Questions
- Which tokens attend most strongly to "forge"?
- How does punctuation affect focus?
Reflection
Reflection — Focus and Forgetting
What do you pay attention to when you think—and what do you ignore? Write 5–7 sentences describing one domain where changing your focus could improve clarity or creativity.