The prototype always works on your laptop. You pass a curated input to a large managed model, wait a few seconds, and watch a flawless response print to the console. It feels complete. Then it meets a live environment. The latency that felt acceptable locally becomes real friction under traffic. The cost per request, negligible in development, compounds into a serious operational expense. The model starts failing in ways the demo never exposed β dropping context, hallucinating details, or breaking structurally when handed messy, out-of-bounds user data.
This rung is production engineering. Moving an agentic system across the threshold means replacing generic assumptions with specific constraints. You stop reading benchmark leaderboards and start evaluating models against the exact operational profile of your task. A heavy reasoning step might need a flagship model; a fast routing layer might run better on a specialised small one. You adapt the model to the workload β deciding where to inject context and where to touch the weights. Then you serve that compute economically.
Production is an unforgiving filter. Cost, latency, and rigorous evaluation decide whether earlier architectural planning survives contact with reality. An elegant orchestration pattern is useless if running it consumes your margin, or if the time-to-first-token stretches past human patience. Getting to production means defining the exact bounds of acceptable performance and holding the deployment to them.
- Stop ranking LLMs, start profiling them β choose by fit to the task, not a leaderboard.
- A practical checklist for picking an LLM for your feature β the concrete selection pass.
- Fine-tune, RAG, or prompt: which one, and what each costs β how to adapt a model, and the price of each route.
- Where LLM serving costs actually go β the economics of serving under load.
- A 70B model on one GPU: a practical guide to quantization β squeezing real models onto real hardware.