15 posts found
The context window is a budget, not a memory
Bigger context windows did not give models memory. They gave you a larger envelope to fill on every single request — and...
Why models hallucinate, and what actually reduces it
Hallucination is not a glitch that a better model will one day remove. It is what generation does when it has nothing to...
System prompts: the instructions the user never sees
A system prompt sets the rules for everything that follows. Written well it is the cheapest quality improvement availabl...
Structured output beats parsing prose
If your code needs a value from a model, ask for JSON and validate it. Regexing an answer out of a paragraph is a bug wa...
Choosing between a large model and a small one
Most production traffic does not need the largest model available. Routing by task rather than defaulting to the top of...
RAG in one page: retrieve, rank, answer
Retrieval-augmented generation is three steps and a lot of tuning. The architecture is simple; the quality lives almost...
Chunking is the part of RAG nobody tunes
Teams spend weeks on rerankers and leave chunking at the default. It is usually the other way round that pays — how you...
Prompt caching: the cheapest speedup you are not using
If every request begins with the same two thousand tokens of instructions, you are paying full price to resend them. Ord...
Streaming responses without breaking your UI
Streaming makes a slow response feel fast. It also means rendering text that is syntactically incomplete, and handling a...
Rate limits, retries and the backoff you actually need
Every AI feature meets a 429 eventually. Whether that is a blip or an outage depends on retry logic written before you n...
How to evaluate an AI feature before you ship it
You cannot unit test "is this a good answer", but you can build a set of real cases with known-good outputs. An afternoo...
The cost model of an AI feature
Per-token pricing looks trivial until you multiply by retries, conversation history and the context you resend on every...