articles with tag

Tokens & Embeddings: How Models Read

A model never sees your sentence. A tokenizer turns it into numbered pieces, and an embedding turns each number into a point. Here is how that cut works in tiktoken, what GPT-5 and gpt-oss charge per piece, and why the count is the bill.

Read More