What a language model actually does

A large language model is trained on a huge amount of text to do one thing: predict what comes next. Given the start of a passage, it estimates which piece of text is most likely to follow, adds it, and repeats. Chat assistants are language models further trained to follow instructions and hold a conversation.

Three ideas explain most of their behaviour:

• Tokens. Models read and write text in chunks called tokens — often a word or part of a word. Prices and limits are counted in tokens, not words.
• The context window. Everything the model can take into account at once — your instructions, any documents you paste in, and the conversation so far — must fit in its context window. It has no memory of other conversations unless an application supplies them.
• No lookup. The model does not search a database of facts when it answers. It generates text that fits the patterns it learned. That is why it can write fluently about almost anything, and also why it can state something false with total confidence.

The practical consequence: a model can only work with what it learned in training plus what is in the context window. The single most effective way to improve an answer is usually to give it the information it needs rather than hoping it already knows.