Understanding Tokens and Context Windows in Language Models
Explains how tokens work as the measurement system for language models, affecting both processing and pricing.
Transcript:
When we start dealing with language models like ChatGPT or Claude, one of the important things we need to manage is something called a context window. A context window represents the amount of information that the language model can handle at any one time. Now the context window is actually measured in what are called tokens. And tokens are kind of like the call measurement system for the way that we deal with language models. Generally tokens are about four characters long and it’s important to understand them because they can also affect pricing that you pay when you’re dealing with language models. What you’re seeing in front of you is a token counting system that I’ve built that allows me to put in some text and this is just some text off our website and it then goes through and derives the number of tokens that are used. And you can see in most cases, if breaking the words down into small chunks are about four characters but there are a couple of things I wanted to highlight for you because one of the ways that a language model works is by finding patterns in words. So if you look at the word information here, you can see that rather than just breaking it up into chunks of four characters, if identified common patterns, the letters I N, the letters F-O-R-M, A-T and I-O-N. And so the way that a language model breaks up the words makes it easier for it to then be able to interpret what you’re asking and be able to look for patterns to be able to give you a meaningful answer. If you’d like to know more about this, let me know.