7.4 Interactive Game: Training a Language Model
Introduction
Language models don't memorize phrases. Instead, they adjust millions of numbers (parameters) based on countless related examples. Each time they process new data, they slightly modify these internal values to improve future predictions.
🏢
Activity
Training a Language Model
Language model training allows AI to discover patterns in text without explicit rules, only through repeated exposure to related examples.
How to Explore It
- 📝 Goal: Predict missing, next, or incorrect words in a thematic paragraph.
- 🧠 Mechanics: A small neural network adjusts its values after each prediction.
- 📊 Learning: Network nodes change color based on their values (blue = negative, white = zero, red = positive).
- 🎯 Progress: As you advance through the paragraph, predictions become easier thanks to accumulated context.
What to watch for:
The model explores different textual contexts, adjusting its internal parameters. With each processed example, its predictions gradually improve. You'll understand how accumulated context makes predictions more accurate.
Interactive Demonstration
Language Model Trainer
🧠 Neural Network
Negative
Zero
Positive
Examples processed
0
Correct
0
Errors
0
Paragraph progress
1 / 8
📖 Accumulated context
Context will appear here as you progress...
🎯 Complete the sentence
Core Concepts
How Does the Model Learn?
The model adjusts its parameters iteratively:
- Presentation: A sentence is shown with a word to predict
- Prediction: The user (or model) chooses an answer
- Evaluation: The response is compared to the correct one
- Adjustment: Internal values are modified based on the result
- Accumulation: Context from previous sentences facilitates future predictions
Training Modes
- Missing word: Fill in the blank in a sentence (BERT-style)
- Next word: Predict what word comes next (GPT-style)
- Correction: Identify and correct an incorrect word