Quick Check: From Last Time

Before we move, let's try those questions from the last session on Wednesday, July 08, 2026, about word embeddings.

(No right or wrong — these just warm up our intuition. A full recap is on the next slide.)

Want to refer back? Last session's deck (on GitHub Pages): → Embeddings deep-dive

How Meaning Works: Embeddings → Similarity

A full t-SNE example from scratch. We see how a word's vector captures meaning, how the dot product combines two vectors, and how cosine similarity measures how alike their meanings are.

Dataset — 5 words with 3D embeddings

Visual intuition: dog & puppy point in nearly the same direction · man & woman point in similar directions · tree points in the opposite direction from dog.

Step 1 — Cosine similarity formula

cos(A,B) = (A·B) / (|A| × |B|)

A·B = dot product (multiply matching numbers, add them).   |A| = length (Euclidean norm) of A.

Step 2 — Vector Norms

Step 3 — Dot products

dog·puppy = 30+72-1 = 101  |  dog·man = 36-18+8 = 26  |  dog·woman = 42+27+9 = 78

dog·tree = -42-72-3 = -117  |  puppy·man = 30-16-8 = 6  |  puppy·woman = 35+24-9 = 50

puppy·tree = -35-64+3 = -96  |  man·woman = 42-6+72 = 108  |  man·tree = -42+16-24 = -50  |  woman·tree = -49-24-27 = -100

Step 4 — Cosine similarities

Interpretation: dog–puppy 0.98 (almost identical) · man–woman 0.90 (both humans) · dog–tree −0.98 (opposite meaning) · dog–woman 0.61 (both living things) · puppy–man 0.06 (barely related).

Your Turn: Calculate Similarity Yourself

Use cos(A,B) = (A·B) / (|A|×|B|). For each pair: (1) dot product, (2) each vector's length, (3) divide. Then check against the matrix on the previous page.

Bonus: Pick your own pair from the dataset, or invent two 3D vectors and compare answers with a neighbour.

What We'll Build in 2 Hours

No prior ML experience needed. Just Python basics and curiosity.

One-Time Setup

Open the Colab Notebook or your local Python environment. Run these two blocks:

# Import everything we need
import re, math
from collections import defaultdict, Counter
import numpy as np
import matplotlib.pyplot as plt
import torch, torch.nn as nn
from torch.nn import functional as F
import tiktoken

# Check for GPU
device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(f"Running on: {device.upper()}")

How Do We Turn Text Into Numbers?

Computers don't understand words. They understand numbers. Tokenization is the bridge.

Three Ways to Split Text

BPE (Byte-Pair Encoding) is what GPT-4, Gemma, LLaMA, and Claude all use.

How BPE Works (Simple Version)

Imagine you're a toddler learning to read. You start with letters, then notice some pairs appear together a lot.

BPE in Action — Example

Tracing BPE on a tiny corpus: "sunny rainy sunny". Watch letters merge into subwords, step by step. (</w> marks the end of a word.)

Each step merges the most frequent neighbour pair. In just 5 merges, "sunny" becomes a single token — so the model treats it as one meaningful unit instead of six separate letters.

BPE in 4 Functions

We only need 4 small functions to build BPE from scratch.

def get_vocab(corpus):
    vocab = {}
    for word in corpus.lower().split():
        tokenized = " ".join(list(word)) + " </w>"
        vocab[tokenized] = vocab.get(tokenized, 0) + 1
    return vocab

def get_stats(vocab):
    pairs = defaultdict(int)
    for word, freq in vocab.items():
        symbols = word.split()
        for i in range(len(symbols) - 1):
            pairs[(symbols[i], symbols[i+1])] += freq
    return pairs

def merge_vocab(pair, vocab):
    a, b = pair
    merged = a + b.replace("</w>", "")
    pattern = " ".join([a, b])
    new_vocab = {}
    for word, freq in vocab.items():
        new_vocab[word.replace(pattern, merged)] = freq
    return new_vocab

Let's Train It!

We'll feed BPE sentences about African farming and watch it learn.

CORPUS = "Cassava is a major food crop in Nigeria and across sub-Saharan Africa. Farmers in Kenya grow maize beans and sweet potatoes. The cocoa industry in Ghana and Ivory Coast produces most of the world's chocolate. Climate change affects rainfall patterns across East Africa. Drought resistant crops like millet and sorghum are important for food security. Smallholder farmers use sustainable farming practices to improve crop yields."

# Train to 50 total tokens (starts from ~30 letters)
merges, final_vocab, _ = bpe_train(CORPUS, vocab_size=50)

# Sample output you'll see:
Merge 1: 'a' + '</w>'  →  "a</w>"
Merge 5: 't' + 'h'    →  "th"
Merge 10: 'in' + 'g'   →  "ing"

Then we tokenize a new sentence:

test = "Farmers grow cassava in Nigeria"
tokens = tokenize_bpe(test, merges)
print(tokens)  # → ['Farm', 'ers', 'grow', 'cass', 'ava', 'in', 'Nig', 'eri', 'a']

Notice how "cassava" splits into cass + ava — the algorithm learned that "cass" and "ava" are useful subwords!

Switch to a Production Tokenizer

Our custom BPE works, but it only has 50 tokens. Real LLMs use 100,000+ tokens. Let's use tiktoken — the tokenizer behind GPT-4.

# Tiktoken has ~100K tokens trained on billions of pages
enc = tiktoken.get_encoding("cl100k_base")

# Our tiny story for training
raw_text = """Jide was hungry so she went looking for food. She checked the market, but it was closed. Then, she found a small cafe and ordered some rice and water. She was very happy to finally eat!"""

# Encode: text → numbers
data = torch.tensor(enc.encode(raw_text), dtype=torch.long)

print(f"Characters: {len(raw_text)} → Tokens: {len(data)}")
# Output: "Characters: 188 → Tokens: 43"

# Decode a token back to prove it works
print(enc.decode([data[0].item()]))  # → "Jide"

43 tokens to represent 188 characters. That's the power of a good tokenizer — it compresses text efficiently.

Embeddings: Words as Vectors

A number alone (token ID) has no meaning — "397" and "401" don't tell us the words are related. Embeddings fix this by mapping each token to a vector of numbers where similar words live close together.

Embedding for "king" → [0.23, -1.45, 0.67, ..., 0.12]  (32 numbers)
Embedding for "queen" → [0.25, -1.40, 0.62, ..., 0.15]  (very similar!)
Embedding for "apple" → [-0.98, 2.10, -0.33, ..., 0.88]  (very different)

Last session you explored embeddings in depth — t-SNE maps and cosine similarity. Today we put them to work: those same vectors become the model's memory, learned live as it trains.

Creating Training Data: The Sliding Window

A language model's only job: predict the next token. We create flashcards: "Given these tokens, what comes next?"

block_size = 8  # How many tokens the model sees at once
batch_size = 4  # How many examples we process together

def get_batch():
    ix = torch.randint(len(data) - block_size, (batch_size,))
    x = torch.stack([data[i:i+block_size] for i in ix])  # Context
    y = torch.stack([data[i+1:i+block_size+1] for i in ix])  # Target
    return x.to(device), y.to(device)

The model sees 8 tokens at a time and must predict token #9. We make thousands of these from our tiny story.

The Tiny Brain: Our SLM Architecture

A miniature version of what powers ChatGPT. Three parts:

nn.Embedding(vocab_size, n_embd=32)

nn.Embedding(block_size, n_embd)

nn.Linear(n_embd, vocab_size)

class TinySLM(nn.Module):
    def __init__(self, vocab_size, n_embd):
        super().__init__()
        self.token_embedding_table = nn.Embedding(vocab_size, n_embd)
        self.position_embedding_table = nn.Embedding(block_size, n_embd)
        self.lm_head = nn.Linear(n_embd, vocab_size)

    def forward(self, idx, targets=None):
        B, T = idx.shape
        tok_emb = self.token_embedding_table(idx)
        pos_emb = self.position_embedding_table(
            torch.arange(T, device=device))
        x = tok_emb + pos_emb
        logits = self.lm_head(x)
        if targets is None:
            return logits, None
        B, T, C = logits.shape
        loss = F.cross_entropy(logits.view(B*T, C), targets.view(B*T))
        return logits, loss

The two embedding tables are the heart of the model — they store the meaning of every word, exactly the vectors you explored last session. Training is just the model learning better embeddings so it can predict what comes next.

Total: ~6.5 million parameters. Tiny by today's standards (GPT-4 has trillions), but enough to learn patterns!

Training: Becoming Less Wrong

A fresh model guesses randomly. Training fixes that — one tiny adjustment at a time.

optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)

for step in range(600):
    xb, yb = get_batch()                # Grab a batch
    logits, loss = model(xb, yb)         # Forward pass
    optimizer.zero_grad(set_to_none=True)
    loss.backward()                      # Backward pass
    optimizer.step()                     # Update weights
    if step % 100 == 0:
        print(f"Step {step}: Loss = {loss.item():.4f}")

Watch the loss go from ~10 (random guessing) down to ~1.5 (learning patterns).

Let It Speak!

After training, our tiny model can generate text. We give it a starting word and ask: "What comes next?"

def generate(self, idx, max_new_tokens, temperature=1.0):
    for _ in range(max_new_tokens):
        idx_cond = idx[:, -block_size:]         # Only use last 8 tokens
        logits, _ = self(idx_cond)               # Predict
        logits = logits[:, -1, :] / temperature  # Scale by temperature
        probs = F.softmax(logits, dim=-1)        # Convert to probabilities
        idx_next = torch.multinomial(probs, 1)   # Sample
        idx = torch.cat((idx, idx_next), dim=1)  # Append
    return idx

Temperature trick: Lower = more focused/repetitive, Higher = more creative/risky. Try temperature=0.8!

Making Our Model Smarter

Our first model was tiny (32-dim embeddings, 8-token context, 600 steps). Let's level up.

Result: A model that generates much more coherent text. Try it in Colab — change these 3 numbers and watch the difference!

Scale the recipe further: The same architecture at a larger size reaches 12,937,781 parameters — more capacity means more patterns it can capture. (Our BiggerSLM uses ~6.5M; bigger still = smarter still.)

What You Learned Today

Thank You!

Questions? Let's discuss.