The unit a model reads
People read letters and words. A language model reads tokens. A token is a chunk of text that the model treats as a single unit. Depending on the text, a token might be a whole common word like “the,” a piece of a longer word, a number, a punctuation mark or a space attached to a word.
For example, a short common word is usually one token, while a long or rare word may be split into several pieces, much like breaking “unbelievably” into “un,” “believ” and “ably.” The exact splits depend on the particular system, so the example is illustrative and not what any specific model does.
Why not just use words or letters?
Using whole words would require an enormous vocabulary, and the model would struggle with new words, names, typos and compound terms. Using single letters would make every sentence very long and harder to learn from. Tokens are a middle path: a fixed list of common chunks, learned from text, that can combine to spell nearly anything. A model has a vocabulary of such pieces, often numbering in the tens of thousands or more, and each is mapped to a list of numbers the network can work with.
This conversion happens before the model does anything else. Your prompt is cut into tokens, the model processes them, and then it produces new tokens one at a time, as described in how language models predict text. Finally the tokens are turned back into readable text.
Rough sizes
For ordinary English, a token is often somewhat shorter than a word, so a passage tends to contain a few more tokens than words. A commonly cited rule of thumb is that a token is about three-quarters of a word, but that is only an approximation. It varies by tool and by text.
- Text in other languages, especially those with different writing systems, can use noticeably more tokens for the same meaning.
- Code, numbers, unusual names and text with special characters often use more tokens than plain prose.
Many services provide a token counter, which is the reliable way to know how long something is for a given model.
What a context window is
The context window is the maximum number of tokens a model can consider at once. It includes everything the model is working with in a single exchange: your instructions, any documents you pasted in, the earlier turns of the conversation, and the text the model is generating in reply.
Think of it as a desk of fixed size. Only what fits on the desk can be looked at while writing. Material that does not fit is not partly visible; it is simply not available to the model for that response.
Window sizes differ widely between models and have grown considerably over the years, from a few thousand tokens in earlier systems to much larger figures in recent ones. Larger is not automatically better, though, and the figure for any specific product can change, so check the documentation for the tool you use.
What this means in practice
Long chats can lose earlier details. In a very long conversation, older messages may be cut off or summarized to make room, so the chatbot can seem to forget something you said at the start. Restating key points helps.
Long documents may not fit. If you paste a book chapter or a lengthy report, part of it may be rejected or ignored. Splitting the material into sections and working through them is a common workaround.
Output length is limited too. Because replies also count toward the window, there is a cap on how long a single answer can be.
More text can mean worse attention to details. Even when everything technically fits, models can pay less reliable attention to material buried in the middle of a very long input. Putting the most important instructions and facts clearly, and not burying them, tends to help.
Cost and speed. Many paid services charge by the number of tokens processed. Longer inputs usually mean more computing work, slower responses and, where pricing is per token, higher costs. The computing behind this runs in data centers, which is one reason efficiency matters.
Why the limit exists
The attention mechanism inside most current models compares tokens with one another, so the work required grows quickly as the passage gets longer. Handling more text needs more memory and computing power. Researchers keep developing ways to extend windows more efficiently, but some limit always remains.
A context window is not long-term memory
The window holds what the model can use right now. It does not mean the system has permanently learned your information. The model’s general knowledge is stored in its trained parameters, which stay fixed during a chat. Some products add separate memory features that save notes between sessions, and these work differently and have their own privacy implications, so it is worth understanding what a tool stores. Sensitive details are best kept out of prompts, as discussed under responsible AI use.
Working with the limits
- Be concise. Remove irrelevant text before pasting.
- Put key instructions where they are easy to find, and repeat them after long material if needed.
- Break large jobs into steps and carry the results forward in short summaries.
- Start a new conversation for a new topic, rather than stretching one thread indefinitely.
- Remember that a limited window is one reason answers can be incomplete or wrong, along with the factors behind AI hallucinations.
The takeaway
Tokens are the pieces of text that language models work with, and the context window is the cap on how many of them a model can handle at once. Knowing this explains many everyday oddities: why a long chat drifts, why a pasted document is partly ignored, and why a request to write something very long may be cut short.