Tokenization & Embeddings
BPE, subword vocabularies, and how tokens become vectors — the input side of every LLM.
Introduction
Tokenization and embeddings are the first translation layer between human text and neural computation. A language model never sees words directly. It sees integer token ids, then dense vectors, and every later transformer block operates on those vectors.
In interviews, this topic separates API familiarity from systems understanding. A strong answer explains why modern models use subword tokens, how a tokenizer learns a vocabulary, how token ids index an embedding matrix, and why token count controls cost, latency, and context limits.
This lesson covers the practical path from raw text to vectors: byte-level BPE and related tokenizers, special tokens such as beginning of sequence, end of sequence, and padding, the embedding matrix learned during training, and the geometry behind cosine similarity.
The same concepts also power retrieval. Sentence and document embeddings are stored in vector databases for RAG, while prompt tokenization determines how much retrieved context fits in the model window. If you can reason about both, you can design cheaper, more reliable LLM systems.
Where this shows up in production
Tokenization and embeddings show up in every production LLM feature.
- Chat products estimate prompt size, output room, and price in tokens, not words.
- Long documents must be chunked by token count so they do not overflow the context window.
- Multilingual text, emojis, logs, and code can consume more tokens than expected.
- Embedding models turn chunks into vectors that can be searched in a vector database.
- RAG quality depends on both chunk tokenization and the embedding model used for retrieval.
- Model upgrades can require tokenizer compatibility checks and embedding re-indexing.
Learning Objectives
Define what a token is and why modern LLMs usually use subword tokens.
Explain how BPE learns a vocabulary by repeatedly merging frequent adjacent pairs.
Distinguish byte-level BPE, WordPiece, and SentencePiece at a practical level.
Trace how tokens become ids, special tokens, and embedding vectors.
Use token-count heuristics to reason about pricing, context limits, truncation, and prompt budgets.
Compare static, contextual, and sentence or document embeddings for retrieval and RAG.
Theory & Concepts
A token is the model input unit
A token is one item from the tokenizer vocabulary. It may be a whole word, part of a word, punctuation, whitespace, a byte sequence, or a common code fragment. The model input is not text; it is a sequence of token ids, one integer per token.
Subword tokens are the practical compromise between words and characters. Word-level tokenizers break on rare words, names, typos, and morphology. Character-level tokenizers produce very long sequences. Subword tokenizers keep common words whole but split rare words into reusable pieces, giving a manageable vocabulary and shorter sequences.
BPE learns vocabulary by merging frequent pairs
Byte-Pair Encoding starts from small symbols and repeatedly merges the most frequent adjacent pair in the training corpus. If the pair t plus h appears often, it may become th. Later th plus e may become the. After many merges, the tokenizer has a vocabulary containing single symbols, common subwords, whole common words, punctuation patterns, and code fragments.
At inference time the learned merge rules are applied greedily and deterministically. The tokenizer does not understand meaning; it follows a learned compression scheme that tends to keep frequent patterns short and splits uncommon patterns into smaller pieces.
Byte-level BPE, WordPiece, and SentencePiece
Byte-level BPE starts from bytes instead of a fixed set of characters. This gives full coverage: any Unicode text, emoji, binary-looking log fragment, or unusual symbol can be represented without an unknown token. The tradeoff is that some non-English text and unusual strings can tokenize into more pieces.
WordPiece, used by BERT-style models, is also a subword method but selects vocabulary entries using a likelihood-oriented objective rather than pure pair frequency. SentencePiece is a tokenizer toolkit that treats text as a raw stream and can train BPE or unigram tokenizers without relying on spaces, which makes it useful across languages.
Token ids and special tokens
After segmentation, every token is mapped to an integer id by a vocabulary table. The sequence of ids is what the model receives. The same visible text can produce different ids under different tokenizers, which is why tokenizer and model versions are tied together.
Special tokens are reserved ids with control meaning. A beginning-of-sequence token can mark the start of input, an end-of-sequence token can mark completion, and a padding token can fill shorter examples in a batch. Training and serving code must handle special tokens and padding masks consistently.
The embedding matrix maps ids to dense vectors
The embedding layer is a learned matrix with one row per vocabulary id and one column per hidden dimension. Looking up token id 824 means selecting row 824 from that matrix. The result is a dense vector, and a prompt becomes a sequence of dense vectors before it enters the transformer.
These vectors are learned during training. Tokens that appear in similar contexts tend to receive related representations, so the geometry carries semantic information. Cosine similarity measures the angle between vectors and is commonly used when comparing embeddings because it focuses on direction rather than raw magnitude.
Static, contextual, and retrieval embeddings
Static embeddings such as word2vec and GloVe assign one vector to a word type, so bank has the same vector in river bank and investment bank. Transformer models produce contextual token states, so the representation of bank can change based on neighboring tokens.
Sentence and document embeddings compress a span of text into one vector for search, clustering, or retrieval. These are usually produced by embedding models trained for semantic similarity. In RAG, those vectors are indexed in a vector database, and the most similar chunks are retrieved before generation.
Architecture Diagram
The diagram separates deterministic preprocessing from learned parameters. The tokenizer and vocabulary are fixed for a model release: they decide the token boundaries and ids. The embedding matrix is learned during training and maps those ids into the hidden vector space used by the transformer.
The vector database branch is conceptual. Internal token embeddings feed the transformer, while RAG systems usually store sentence or document embeddings produced by a retrieval embedding model. Both rely on dense-vector geometry, but they are trained for different objectives and should not be treated as interchangeable.
The dashed retrieval edge means retrieved text must still be placed into the prompt and tokenized before generation. Retrieval reduces the amount of raw context the model must read, but the final request is still limited and priced by tokens.
Request Flow
- 1
1. Receive raw text
The system starts with user text, a document chunk, a tool result, or source code. Before modeling, the text is interpreted as a sequence of bytes or Unicode characters, including whitespace and punctuation that may affect token boundaries.
- 2
2. Apply tokenizer rules
The tokenizer applies its learned vocabulary and merge rules. Frequent substrings are kept as larger tokens, while rare words, names, non-English text, or unusual code patterns are split into smaller pieces.
- 3
3. Map tokens to ids
Each token is replaced with its integer vocabulary id. The serving request may also add beginning-of-sequence, end-of-sequence, separator, or role-control tokens depending on the model and chat format.
- 4
4. Pad or truncate for batching and limits
Training batches often pad shorter sequences so tensors have the same length. In serving, systems count tokens before sending a request and may truncate history, summarize old turns, or drop low-value context to stay within the model window.
- 5
5. Look up rows in the embedding matrix
Every token id selects one row from the embedding matrix. This converts a sequence of integers into a sequence of dense vectors with the model hidden size.
- 6
6. Add position and contextualize
The model injects positional information and then passes vectors through transformer blocks. After attention layers, token representations become contextual, so a token vector reflects both the token identity and surrounding text.
- 7
7. Produce retrieval embeddings when needed
For search, a separate embedding model converts each chunk or query into a sentence or document vector. Those vectors are normalized or otherwise prepared for similarity search, then stored with metadata in a vector database.
- 8
8. Retrieve and fit context into the prompt
At query time the system embeds the query, retrieves similar chunks, and adds selected text to the prompt. The final prompt still has to be tokenized, counted, and budgeted against the context window and expected output length.
Deep Dive
Why not tokenize by words
A word tokenizer needs a huge vocabulary to cover names, compounds, misspellings, languages with rich morphology, and code identifiers. Unknown words become a serious failure mode. It also handles punctuation and whitespace poorly unless many special cases are added.
Subword tokenization avoids most unknowns while keeping sequences much shorter than characters. It can represent tokenization as token plus ization, or split a rare identifier into meaningful fragments. The result is not perfect semantics, but it is a robust engineering compromise.
Why token count controls product cost
LLM providers price and limit requests in tokens because tokens drive computation. Input tokens consume context window space and prompt-processing compute. Output tokens require iterative generation work and usually affect latency directly.
A rough English heuristic is that one token is about four characters or about three quarters of a word. This is only a planning estimate. Non-English text, emojis, tables, dense punctuation, and source code can use more tokens per visible character, so production systems should count with the exact model tokenizer.
Embedding geometry and cosine similarity
An embedding vector is useful because distance in the vector space often correlates with semantic relatedness. Queries about password reset should land near documents about account recovery, even if the exact words differ.
Cosine similarity compares vector directions. It is popular in retrieval because two vectors can point in a similar semantic direction even if their magnitudes differ. Many vector databases optimize for cosine similarity, dot product, or Euclidean distance, and the embedding model documentation tells you which metric to use.
Tokenizer choices affect truncation quality
When a prompt is too long, truncation is not just removing characters. It removes tokens, and token boundaries can split words, code, or structured data in awkward places if truncation is naive.
Good systems truncate at semantic boundaries: whole messages, sections, JSON fields, paragraphs, or document chunks. They reserve output space first, then allocate the remaining token budget to system instructions, user input, conversation history, retrieved chunks, and tool results.
Production Considerations
Use the exact tokenizer for budgeting
Do not estimate production limits with word counts alone. Use the tokenizer for the target model to count messages after chat formatting, tool schemas, retrieved chunks, and hidden control tokens. Keep a safety margin for output and for provider-specific wrapping.
Version tokenizers and embedding models
Tokenizer changes can alter token ids and prompt length. Embedding model changes can alter vector dimension, distance distribution, and nearest-neighbor rankings. Store model name, tokenizer version, embedding dimension, and normalization policy with every indexed corpus.
Design truncation as a product behavior
A system that silently drops the wrong context can become incorrect or unsafe. Prefer explicit policies: preserve system instructions, keep recent user intent, rank retrieved chunks, summarize old history, and log how many tokens were removed from each category.
Plan for multilingual and code-heavy workloads
The four-character heuristic is English-centric. Some languages and many code or log formats tokenize less efficiently. Capacity planning, chunk size, and cost estimates should be based on representative traffic, not only English prose examples.
Interview Perspective
What interviewers look for
- ✓A precise explanation that models consume token ids, not words or raw strings.
- ✓Correct understanding of BPE-style vocabulary learning and why subwords are the dominant compromise.
- ✓Ability to connect token count to pricing, context limits, latency, and truncation behavior.
- ✓Clear separation between token embeddings inside a transformer and sentence or document embeddings used for retrieval.
Alternative designs
Word-level tokenization
A word-level tokenizer is easy to explain and can be efficient for common words, but it fails on rare words, names, misspellings, many languages, and code. It also requires an enormous vocabulary or an unknown-token escape hatch. Subword tokenization is more robust.
Character-level tokenization
Character-level tokenization has tiny vocabulary and full coverage, but sequences become much longer. Longer sequences increase attention cost, reduce effective context, and make it harder for the model to learn higher-level units. It is simple but often too expensive for general LLMs.
Likely follow-up questions
How does BPE handle a word it has never seen before?
BPE breaks the word into known subword units by applying learned merge rules. With byte-level BPE, it can always fall back to byte tokens, so there is no unknown token for arbitrary input. The model may still understand rare strings poorly, but they remain representable.
Why does a 10 page English document and a 10 page code file not necessarily cost the same?
Cost follows token count, not page count. Code has punctuation, indentation, rare identifiers, symbols, and short fragments that may tokenize into many pieces. Some non-English text and mixed-format logs also tokenize less efficiently than plain English prose, so exact tokenizer counting is required.
What is the difference between contextual token embeddings and retrieval embeddings?
Contextual token embeddings are internal transformer states that change with surrounding tokens and are used by the model to continue computation. Retrieval embeddings are usually one vector per sentence, passage, or document, trained so semantically similar text is close for search. They serve different objectives.
Common mistakes
- ×Saying tokens are always words, when many tokens are subwords, spaces, punctuation, or bytes.
- ×Using word count as an exact cost estimate instead of counting with the model tokenizer.
- ×Treating word2vec-style static embeddings and transformer contextual states as the same thing.
- ×Assuming a vector database stores the chat model internal token embeddings rather than embeddings from a retrieval model.
Visual Learning
Tokenization granularity tradeoffs
| Granularity | Strength | Weakness | Typical use |
|---|---|---|---|
| Word | Short sequences for common words | Large vocabulary and unknown-word problems | Simple NLP baselines |
| Character | Tiny vocabulary and full coverage | Very long sequences and higher attention cost | Specialized models or fallback logic |
| Subword | Robust coverage with moderate sequence length | Token boundaries are not always intuitive | Modern LLM tokenizers |
Embedding types
| Type | Representation | Context sensitivity | Common use |
|---|---|---|---|
| Static word embeddings | One vector per word type | No, same vector in every sentence | Classic NLP features and similarity |
| Transformer token states | One vector per token position | Yes, changes with neighboring tokens | Internal model computation |
| Sentence or document embeddings | One vector per text span | Usually trained for span-level meaning | Vector search, clustering, and RAG |
Tokenizer families
| Tokenizer | Core idea | Coverage behavior | Where it appears |
|---|---|---|---|
| BPE | Merge frequent adjacent pairs | Depends on starting symbols | GPT-style tokenizers |
| Byte-level BPE | Run BPE from byte symbols | Can represent arbitrary input | Many modern generative models |
| WordPiece | Choose subwords by likelihood benefit | Uses known pieces and continuation markers | BERT-style encoders |
| SentencePiece | Train from raw text without space pre-tokenization | Works well across languages | Multilingual and encoder-decoder models |
Decision guide
How to use tokenization and embeddings in system design
Use the exact model tokenizer whenever you budget prompts, chunk documents, or estimate cost. A rough heuristic is useful for early planning, but production code should count final formatted requests, including system messages, tool schemas, retrieved context, and expected output room.
Use semantic chunking by token count for RAG. Choose chunk sizes that preserve meaning, include small overlaps when needed, and avoid cutting through tables, JSON objects, code blocks, or policy statements. Store the token count with each chunk so retrieval can fit a bounded prompt.
Use a retrieval embedding model for vector search, not the chat model hidden states. Match the vector database metric to the embedding model recommendation, and re-index when you change embedding model, dimension, normalization, or chunking policy.
Use truncation policies that protect correctness. Keep system instructions and current user intent, rank retrieved chunks by relevance, summarize old conversation when possible, and log what was removed so quality issues can be debugged.
Hands-on Examples
Estimate a prompt budget before calling a model
This quick estimator uses the English planning heuristic, then reserves output tokens before deciding whether the prompt fits. Production code should replace the heuristic with the exact tokenizer for the selected model.
prompt_budget.py
def rough_english_tokens(text):
by_chars = round(len(text) / 4)
by_words = round(len(text.split()) / 0.75)
return max(1, max(by_chars, by_words))
def budget_prompt(system_text, user_text, retrieved_chunks, max_context, expected_output):
prompt_text = system_text + "\n" + user_text + "\n" + "\n".join(retrieved_chunks)
prompt_tokens = rough_english_tokens(prompt_text)
available_for_prompt = max_context - expected_output
return {
"prompt_tokens_estimate": prompt_tokens,
"available_for_prompt": available_for_prompt,
"fits": prompt_tokens <= available_for_prompt,
"tokens_to_remove": max(0, prompt_tokens - available_for_prompt),
}
chunks = ["Account recovery policy for locked users.", "Password reset audit requirements."]
result = budget_prompt("Answer with citations.", "How do I reset access?", chunks, 4096, 512)
print(result)Simulate one BPE merge step
This tiny example shows the training idea behind BPE: count adjacent pairs and merge the most frequent one. Real tokenizers add normalization, byte handling, special tokens, and many thousands of merge steps.
bpe_merge_demo.py
from collections import Counter
def adjacent_pairs(tokens):
counts = Counter()
for token in tokens:
for index in range(len(token) - 1):
pair = (token[index], token[index + 1])
counts[pair] += 1
return counts
def merge_pair(token, pair):
merged = []
index = 0
while index < len(token):
if index < len(token) - 1 and (token[index], token[index + 1]) == pair:
merged.append(token[index] + token[index + 1])
index += 2
else:
merged.append(token[index])
index += 1
return merged
corpus = [list("lowest"), list("lower"), list("newest"), list("newer")]
best_pair = adjacent_pairs(corpus).most_common(1)[0][0]
updated = [merge_pair(token, best_pair) for token in corpus]
print("best pair", best_pair)
print(updated)Quiz
0/6 answered
1.What does an LLM consume after tokenization?
2.Why do modern LLMs usually use subword tokens instead of pure word tokens?
3.What is the core training idea of BPE tokenization?
4.Why does token count matter in production LLM systems?
5.What does the embedding matrix do?
6.Which statement best describes retrieval embeddings in RAG?
Flashcards
Cheat Sheet
Tokenization and embeddings cheat sheet
Text to ids
- Raw text enters the tokenizer.
- The tokenizer applies vocabulary and merge rules.
- Tokens map to integer ids.
- Special tokens may mark boundaries, roles, separators, end of sequence, or padding.
- Token ids index rows in the embedding matrix.
- The model receives dense vectors, not words.
Tokenization facts
- A token is not necessarily a word.
- Subword tokenization balances vocabulary size and sequence length.
- BPE learns by merging frequent adjacent pairs.
- Byte-level BPE can represent arbitrary input through byte fallback.
- WordPiece and SentencePiece are related subword approaches used by many encoder and multilingual models.
- Tokenizer and model versions are linked because ids must match learned embedding rows.
Token budget facts
- Providers price and limit requests in tokens.
- Context windows are measured in tokens.
- A rough English estimate is one token per four characters or about three quarters of a word.
- Non-English text, code, logs, emojis, and punctuation-heavy data can tokenize less efficiently.
- Always reserve output tokens before filling the prompt.
- Truncate by semantic units, not arbitrary characters.
Embedding facts
- The embedding matrix has shape vocabulary size by hidden size.
- Rows are learned during training.
- Similar contexts push representations into useful semantic geometry.
- Cosine similarity compares vector direction and is common in retrieval.
- Static embeddings assign one vector per word type.
- Transformer contextual states change with surrounding tokens.
- Sentence and document embeddings are used for vector search and RAG.
RAG connection
- Chunk documents with token budgets in mind.
- Embed chunks with a retrieval embedding model.
- Store vectors and metadata in a vector database.
- Embed the query, retrieve similar chunks, then place selected text into the prompt.
- The final prompt is still tokenized, priced, truncated, and limited by the generation model context window.
References
- PaperNeural Machine Translation of Rare Words with Subword Units — Sennrich et al.
- PaperSentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing — Kudo and Richardson
- PaperGloVe: Global Vectors for Word Representation — Pennington et al.
- Docstiktoken — OpenAI