LLM Fundamentals
Tokens, context windows, temperature and cost math - the mental model before building AI features.
Before this: Working With AI Coding Agents
Objective
Build a working mental model of large language models: what tokens are and why they matter for budgets, how context windows bound everything, what temperature changes, how model classes trade capability against cost and latency - and compute cost math from per-token pricing without hand-waving.
Vocabulary
The words to use when directing your agent.
- token
- The chunk of text - roughly part of a word - that models actually read and write.
- context window
- The maximum tokens a model can consider at once: your prompt plus its answer.
- temperature
- A setting controlling randomness: low is focused and repeatable, high is varied.
- completion
- The text a model generates in response to input.
- per-token pricing
- Costing based on tokens processed in and generated out - the standard billing shape.
Understand it
Tokens drive both limits and bills
A rough English rule of thumb: one token is about four characters, though it varies by text. Every call charges for input tokens AND output tokens; long conversations re-send growing history each turn, so costs climb non-linearly with chat length. Before building any AI feature, sketch its token math: typical prompt size, typical response size, calls per user action. That arithmetic - price times tokens, summed over usage - is the whole business case.
Temperature, determinism and model choice
Low temperature makes outputs near-deterministic - right for extraction and classification. High temperature adds variety - right for brainstorming, wrong for invoices. Model choice follows the same honesty: larger models generally reason better but cost more and respond slower per token; smaller ones win on cost and latency. Pick per task, not per hype cycle, and record WHY so future-you can revisit.
Core concepts
Text becomes numbers becomes text
Models do not read letters; they read tokens mapped to learned patterns, then predict likely next tokens repeatedly. Everything you love (fluency) and fear (confident nonsense) flows from this mechanism. Outputs are plausible continuations, not database lookups of truth - which is why grounding (coming later) matters.
Context is a budget you spend
Your Level 0 lesson returns with teeth: the context window holds system instructions, conversation history, retrieved documents AND the answer being written. Stuff it carelessly and either requests fail or attention dilutes across junk. Treating context as a scarce budget - measured in tokens, spent deliberately - is the core AI-native engineering skill.
Examples
The same prompt, two temperatures
Ask for expense categories at temperature near zero twice: nearly identical answers - ideal for pipelines. Crank temperature high: varied phrasing each run - delightful in a writing assistant, alarming in accounting software. One dial, two products.
Cost math on paper
Feature draft: summarize each expense note. Assume 200 input tokens and 100 output tokens per call at some provider's published rates. Ten expenses means ten calls; multiply rate by tokens by calls - done, the feature's marginal cost per user action exists ON PAPER before any code. No invented benchmarks required, just arithmetic.
Common mistakes
Common mistake
Assuming outputs are factual because they sound confident.
Fix
Treat every completion as plausible-sounding until grounded against data you control - retrieval and citations arrive in upcoming modules.
Common mistake
Ignoring token counts until an invoice or truncation surprise.
Fix
Log estimated tokens per call from day one; make context spend visible like any other resource.
Working with your AI agent
Have your agent build a tiny estimation helper early - characters divided by four as a starting heuristic, clearly labeled as approximate - and print estimated tokens alongside every prototype call. When choosing between model options, require a written comparison table: capability needs, relative cost class, latency tolerance - reasoning captured, not vibes.
Build it
The module’s real project. Your agent builds; you direct and inspect.
Project: Token Sense
Build Token Sense: a local tool that estimates token counts for pasted text using a stated heuristic, computes cost scenarios from YOUR entered per-token rates, and runs scripted experiments comparing temperature settings through a pluggable adapter - live API optional, mock mode default.
Requirements
- Paste-in text yields estimated token count via documented heuristic (labeled approximate)
- Cost calculator: enter any provider's published per-token rates, see scenario totals for configurable call volumes
- Provider adapter pattern: mock adapter default (deterministic canned responses); real provider only via server-side proxy reading keys from env
- Temperature experiment script: same prompt run at low and high settings, responses logged side by side
- Every call logs estimated input/output tokens - visible context spending
- README explains the heuristic's limits honestly
Context for your agent
Build me a local Token Sense tool. Core pieces: token estimator using a stated characters-based heuristic (clearly labeled approximate), a cost-scenario calculator where I enter arbitrary per-token rates and call volumes, and a provider adapter layer - deterministic MOCK by default, real LLM API only behind a server-side proxy with env-var keys if I choose to enable one. Scripted experiment runner compares low vs high temperature on identical prompts through whichever adapter is active. Log estimated tokens everywhere. No specific vendor assumptions hardwired.
Verify it - twice
Check every item yourself first. Then hand your agent the prompt and compare verdicts.
Verification prompt - paste into your AI agent
Audit my Token Sense build as a skeptical engineer who distrusts hand-wavy AI math. Verify PASS or FAIL with evidence: (1) check my estimator against several texts using its own stated rule - does it obey itself?, (2) recompute three cost scenarios by hand - does my calculator match arithmetic, not marketing?, (3) attempt to reach a REAL provider through any path while in mock mode - prove no accidental credential use or network calls exist, (4) review my README's claimed heuristic limits - are they honest about where the approximation breaks? Then tell me which assumption in my cost math would most embarrass me at a pricing meeting.
Run this against “LLM Fundamentals” after your own checklist pass. Where the agent and you disagree, investigate before moving on.
Review questions
Answer out loud. If you cannot, revisit before moving on.
Explain tokens to a friend using only the words chunk, predict and bill.
Why does a long chat cost more than linear multiplication of one message's price?
Which tasks did YOU decide want low temperature, and why?
You can now
- Reason about tokens, context budgets and cost math concretely
- Choose temperature and model classes per task with recorded reasoning
- Keep provider access abstracted, mocked and credential-safe by default
Up next
Structured Prompts and Output