Tokenizer Artifacts and Gotchas
Lesson 4 of 4 in Tokenization: Text Becomes Tokens.
The tokenizer is invisible right up until it bites. A famous class of failure: ask a model how many of some letter are in a word, and it may answer wrong with total confidence. The reason is not stupidity — it is that the model may never see the letters. If the word arrives as one or two tokens, the letters inside are not part of the input; recovering them requires the model to have memorized each token’s spelling. Character-level questions are asked of a system that reads in chunks. The same mechanism degrades rhyming, anagrams, character-precise editing, and counting anything inside a token.
That is the template for every gotcha in this lesson: behavior that looks like a reasoning bug is often a representation artifact. Learn to ask “what did the model actually see?” before “why did it think that?”
Bar chart with four bars showing illustrative token counts for variants of one word: "unbelievable" as 2 tokens, "Unbelievable" as 2 tokens, "UNBELIEVABLE" as 5 tokens, and the typo "unbelivable" as 4 tokens. The chart illustrates that capitalization and misspellings can multiply token counts.
Three more artifacts earn a permanent place in your debugging checklist.
Whitespace and case are part of the token. In byte-level tokenizers, world and world (leading space) are different tokens, and Hello / hello / HELLO may tokenize completely differently. This is why a trailing space at the end of a prompt can measurably change a completion, and why string-matching your way through model output is fragile.
Multilingual text inflates. A tokenizer trained mostly on English-heavy data learns generous merges for English and stingy ones for everything else, so the same meaning expressed in an underrepresented language or script becomes more tokens. The consequences compound: higher per-request cost, less effective Context window for the same content, and more generation steps — so more latency. The inflation factor is an empirical property of each tokenizer-language pair; measure it on your own traffic rather than trusting any quoted ratio.
Numbers split oddly. Frequency-driven merges treat digit strings like any other text, so 1234567 may split into uneven chunks that align poorly with place value — one reason digit-level arithmetic is hard to learn from text. Some modern tokenizer designs deliberately force digits into single digits or fixed-size groups to make numeric structure regular; the takeaway for you is that numeric formatting (separators, padding, decimal style) changes what the model sees.
The model miscounts letters in a word
It reads tokens, not letters — if the word is one token, its letters were never in the input. Workarounds that exploit the mechanism: split the word into characters yourself (s t r a w b e r r y), or ask for the spelling first so the letters enter the context as visible tokens.
Two “identical” prompts behave differently
Diff them at the token level, not the string level. Trailing whitespace, a leading space, curly versus straight quotes, non-breaking spaces, or different Unicode normalization all produce different token sequences — invisible in most editors, fully visible to the model.
Costs jumped when the product went multilingual
Token inflation: underrepresented languages spend more tokens per unit of meaning under an English-leaning tokenizer. Budgets, context planning, and latency targets calibrated on English traffic quietly break. Re-measure tokens-per-request per language on real traffic.
Arithmetic fails on long numbers
Uneven digit chunking misaligns place value across examples. Formatting numbers consistently helps, as does routing real arithmetic to a calculator or code tool instead of the model’s next-token guesses. (Orchestrating such tool calls is agent territory — our sister AI Agent Academy teaches it; here we stop at the representation.)
A specific rare string makes the model behave erratically
Vocabularies can contain entries that were nearly absent from model training data — historically nicknamed “glitch tokens”. Their embeddings are undertrained, so outputs touching them can be unstable. Rare, but worth knowing when one bizarre string keeps derailing an otherwise sane system.
In production
Tokens are the metering unit of managed LLM platforms: bills, rate limits, and context budgets are all denominated in tokens, and the tokenizer decides how many tokens your text becomes.
AWS
Amazon Bedrock meters input and output tokens separately, and each hosted model counts with its own tokenizer — so the same prompt is a different token count, and a different bill, on different models. Everything you resend on every call (system prompt, retrieved context, few-shot examples) is metered every call; mechanisms like prompt caching discount repeated token prefixes precisely because the meter is tokens, not requests.
Azure
Azure OpenAI in Azure AI Foundry denominates both billing and quota in tokens — rate limits are tokens-per-minute, so a verbose prompt does not just cost more, it burns your throughput ceiling and triggers throttling sooner. Capacity planning starts from measured token counts of real traffic, and token-inflating inputs (multilingual text, ALL-CAPS logs, pretty-printed JSON) shrink how many requests fit in a deployment.
Google Cloud
Vertex AI expresses generative usage and context limits in tokens and exposes a count-tokens API so you can measure a prompt before sending it — the right tool for pre-flight budget checks and for detecting when a template change quietly doubles input size. Multilingual products feel tokenization most here: the same meaning in a less token-efficient language consumes more budget and more context per request.
Tool: Tokenizer Playground — See all of this with your own strings: paste text into the Tokenizer Playground and watch how casing, whitespace, languages, and numbers change the split.