AI Tokenization and Embeddings
Turning text into something a model can learn from happens in two distinct steps: tokenization splits text into pieces, and embeddings map each piece to a vector of numbers that captures meaning.
Step One: Tokenization
Tokenization is the process of breaking a piece of raw text into smaller units called tokens. Depending on the tokenizer, a token might be a whole word, a piece of a word (a subword), or even a single character. Splitting on whitespace is the simplest possible approach, but it struggles with punctuation, made-up words, and languages that don't use spaces between words - which is why most modern systems use subword tokenization instead.
Example
# A very simple whitespace-based tokenizer, just to illustrate the idea.
# Real tokenizers (like Byte-Pair Encoding, used by most modern language
# models) split text into learned subword pieces instead, so that even
# unfamiliar or misspelled words can be represented as a combination of
# familiar smaller pieces.
text = "Tokenizing isn't always straightforward!"
naive_tokens = text.replace("!", " !").replace("'", " '").split()
print("Naive whitespace tokens:", naive_tokens)
# A subword tokenizer might instead produce something closer to:
# ["Token", "izing", "is", "n't", "always", "straightforward", "!"]From Tokens to Numbers: Why IDs Alone Aren't Enough
Once text is split into tokens, each distinct token is assigned an integer ID from a fixed vocabulary - for example, 'cat' might become 342 and 'dog' might become 7891. But those raw IDs carry no information about meaning: nothing about the numbers 342 and 7891 tells the model that cats and dogs are both animals, or that they're more similar to each other than either is to 'umbrella'. A model that only sees arbitrary ID numbers has no way to generalize what it learns about one word to a similar word.
Embeddings: Representing Meaning as Coordinates
An embedding solves this by replacing each token's ID with a dense vector - typically a few hundred numbers - learned so that tokens used in similar contexts end up positioned close together in this multi-dimensional space. During training, the model adjusts these vectors so that words appearing in similar situations (like 'cat' and 'dog', both commonly following 'my pet is a...') drift toward similar coordinates, while unrelated words drift apart.
Example
import numpy as np
# Toy 2D "embeddings" (real ones typically have hundreds of dimensions)
embeddings = {
"cat": np.array([0.9, 0.8]),
"dog": np.array([0.8, 0.9]),
"kitten": np.array([0.95, 0.75]),
"umbrella": np.array([-0.7, 0.2]),
}
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
print("cat vs dog:", cosine_similarity(embeddings["cat"], embeddings["dog"]))
print("cat vs kitten:", cosine_similarity(embeddings["cat"], embeddings["kitten"]))
print("cat vs umbrella:", cosine_similarity(embeddings["cat"], embeddings["umbrella"]))
