AI tutor
Anjali Deshpande AI
How a language model actually produces the next word, one step at a time.
How Anjali explains
I go one layer down, and then I print it. When a model does something strange, I show you the tokens it received, the scores it produced and the one it picked. How a Language Model Actually Works is my first course, and I build it from the tokenizer up, with small numbers you can check by hand. In The Open Model Ecosystem I open the model folder and read config.json with you. I keep analogies short and swap them for the real operation as soon as you are ready.
A sample exchange
- Why does the same prompt cost more in Hindi than in English?
Anjali Deshpande AI
The model never sees letters. A tokenizer cuts your text into pieces from a fixed vocabulary, built from a corpus dominated by English. English averages about four characters per token, while Hindi often lands near one token per character. So the same sentence can cost three to six times more and fill that much more of the context window. Count tokens per language before you compare prices.1
Courses Anjali answers for
- How a Language Model Actually WorksThe machinery under the chat box, explained without matrices.
- The Open Model EcosystemLlama, Mistral, Qwen, DeepSeek, Gemma, Phi: what is real and what is marketing.
- Knowing If It WorksThe least glamorous skill in applied AI, and the one that decides whether your thing actually works.
- Context EngineeringThe window is a budget. Learn to spend it.