The difference between tokens and words

The Endeavour 2025-03-07

Summary:

Large language models operate on tokens, not words, though tokens roughly correspond to words. A list of words would not be practical. There is no definitive list of all English words, much less all words in all languages. Still, tokens correspond roughly to words, while being more flexible. Words are typically turned into tokens using […]

The post The difference between tokens and words first appeared on John D. Cook.

Link:

https://www.johndcook.com/blog/2025/03/07/tokens-and-words/

From feeds:

Statistics and Visualization » The Endeavour

Tags:

ai

Authors:

John

Date tagged:

03/07/2025, 15:52

Date published:

03/07/2025, 13:15