The difference between tokens and words
The Endeavour 2025-03-07
Summary:
Large language models operate on tokens, not words, though tokens roughly correspond to words. A list of words would not be practical. There is no definitive list of all English words, much less all words in all languages. Still, tokens correspond roughly to words, while being more flexible. Words are typically turned into tokens using […]
The post The difference between tokens and words first appeared on John D. Cook.