Term tag: Multilingual AI

Natural Language Processing (NLP)

NLP is the field; large language models are now the answer to most of it. What sixty years of work was for, and which parts are still unsolved.

Tokenization

How text becomes the integers a language model reads: BPE, WordPiece and Unigram, why " strawberry" is one token, and what that breaks.