What Is an Embedding?

A computer only understands numbers. So a language model turns every word (token) into a long list of numbers called a vector. We call that vector the word's embedding.

Embedding for "king"  → [0.23, -1.45, 0.67, ..., 0.12]   (1,152 numbers)
Embedding for "queen" → [0.25, -1.40, 0.62, ..., 0.15]   (almost the same!)
Embedding for "apple" → [-0.98, 2.10, -0.33, ..., 0.88]  (very different)

Vectors & Matrices (with NumPy)

To work with embeddings we use the Python library numpy. It makes working with lists of numbers (vectors) and tables of numbers (matrices) easy.

import numpy as np

# A vector and a 2x3 matrix
v = np.array([7, 3, 1, 4])
M = np.array([[6, 1, 4],
              [9, 0, 2]])

print(v.shape)   # (4,)  -> 4 numbers
print(M.shape)   # (2, 3) -> 2 rows, 3 columns

# Grab one row (one word's embedding) and one column
row_3  = embeddings[2, :]   # 3rd word's vector
col_7  = embeddings[:, 6]   # 7th number across all words

The whole embedding table is just a big matrix: rows = words, columns = meaning-coordinates.

Meaning Lives in Directions

The magic of embeddings: words that mean similar things point in similar directions. "king" and "queen" point almost the same way. "apple" points somewhere else entirely.

A simplified 2D view. In reality the space has 1,152 dimensions!

This is why a model can guess meaning even for words it has rarely seen — their vectors land near related words it already knows.

The Dot Product: Measuring Similarity

How do we measure if two vectors point the same way? With the dot product — we multiply their numbers pair-by-pair and add everything up.

dot_product = u·v = u₁v₁ + u₂v₂ + ... + uₖvₖ

A dot product can grow very large in high dimensions, so we usually normalize it — that leads us to cosine similarity.

Cosine Similarity

Cosine similarity is the dot product divided by the lengths of both vectors. The result is always between -1 and +1, so it is easy to compare.

cosine(u, v) = (u · v) / ( ||u|| × ||v|| )

+1 = same direction, 0 = unrelated, -1 = opposite. Notice even antonyms like "sad/happy" score above 0 — they are still both feelings, so the model groups them loosely.

One Number Isn't Enough

It is tempting to ask "what does the 5th number mean?" In practice, no single number carries a clear meaning. Meaning comes from the whole vector together.

# Pick any 2 dimensions and plot the words
emb.plot_embeddings_dimensions(embeddings, labels,
                               dim_x=432, dim_y=877)

Try different pairs — you will rarely find a single pair that cleanly separates "food" from "technology". That is normal!

Seeing in 2D: t-SNE

t-SNE is a technique that compresses 1,152 dimensions down to 2, while trying to keep nearby words nearby and far words far. The result is a map we can actually look at.

emb.plot_embeddings_tsne(embeddings, labels)

t-SNE is a viewing tool, not the model itself. It helps us see and debug what the embeddings learned.

Try It Yourself (Live)

In today's hands-on part we will run the Google DeepMind notebook and watch embeddings come alive. You will:

• Load 24 real token embeddings from the Gemma model.
• Write a tiny function to pull out one word's vector.
• Compute cosine similarity between word pairs like "king"/"queen".
• Plot single dimensions, then use t-SNE to see the clusters.

No answers are hidden — run the cells, change the words, and discuss what you see. That is where the learning happens!

Let's Dive into the Lab and Practice Some coding. 💻🔥

Key Takeaways

Today you experimented with real embeddings — the same idea that powers the "meaning" inside every language model, including the tiny SLM we built on Wednesday.

Thank you! Questions, debates, and "why does this happen?" moments are very welcome — that is what today is for. 😊