The model nails the demo. You feed it a clean prompt, and the output is plausible. It formats the response correctly. Then you deploy it against your actual production domain—your internal acronyms, your legacy document structures, the edge cases unique to your operation—and the illusion breaks. Latency climbs as you stuff the context window with routing rules. The strict schema degrades into conversational text when the prompt gets too complex.
The natural reflex for engineering teams is immediate: train a custom model. In traditional software, if an upstream library fails your specific constraints, you fork it or write your own. Applied to large language models, that reflex is usually wrong. Sometimes, however, it is exactly right. The difficulty is distinguishing between the two states before committing months of engineering time.
The adaptation ladder
You do not jump straight to training. You climb an adaptation ladder. Each rung buys tighter fit and more control over the model’s behaviour. Each rung also exacts a heavier toll in operational burden, compute cost, and infrastructure complexity.
1. The off-the-shelf frontier model You call an external API. The model is general-purpose. Maintenance is zero; latency and cost are strictly transactional.
2. Prompt and context engineering You shape the input. You provide few-shot examples or rewrite instructions to bound the model’s behaviour. It is cheap to iterate, but you pay for those instructional tokens on every single inference pass.
3. Retrieval-augmented generation (RAG) You ground the model in your own data at runtime. You maintain a vector store and a retrieval pipeline. It solves knowledge gaps without altering the underlying model weights. You trade model-training complexity for data-engineering complexity.
4. Parameter-efficient fine-tuning (PEFT/LoRA/QLoRA) You freeze the base model and adjust a small fraction of the weights via an adapter. The model learns your specific format, tone, or domain syntax. It requires dedicated compute and clean training data, but the resulting adapter is lightweight and can often be trained on a single GPU.
5. Full fine-tuning You update all the weights in a pre-trained model. It deeply embeds domain knowledge and reasoning patterns. The compute cost is high, the dataset requirements are stringent, and the risk of catastrophic forgetting—where the model loses its baseline competence—is real.
6. Custom pre-training You train from scratch on your own tokens. Control is absolute. You own the architecture. The cost is measured in millions of dollars and dedicated cluster time.
Climb only until it fits
You climb this ladder only until the task fits. You stop at the lowest possible rung.
Most enterprise use cases are resolved safely at the third rung. A frontier model combined with rigorous prompt engineering and a reliable RAG pipeline handles the vast majority of reasoning and summarisation tasks. RAG teaches the model what to read; it brings the data to the model.
Adapting the model itself—moving to rung four or beyond—earns its keep only when the gap is durable and expensive to bridge any other way. Fine-tuning teaches the model how to act. You adapt the weights when RAG fails to capture a specific stylistic cadence, when you need strict adherence to a complex output schema without burning thousands of tokens on instructions, or when latency requirements dictate using a smaller, specialised model rather than a large generalist.
Where adaptation loses
Adapting a model is not just a training cost; it is a maintenance commitment.
A fine-tuned or custom model begins going stale the moment training finishes. While a team spends three months curating a dataset and training a specialised 8-billion-parameter model, the baseline capabilities of off-the-shelf frontier models will often leap ahead. You risk shipping a specialised asset that performs worse than the newest general API.
Once you own the weights, you own the serving infrastructure. You must build and maintain pipelines for versioning, evaluation, and deployment. You manage GPU provisioning and handle scaling for your own inference endpoints. For a small team, a low-sensitivity task, or an early architectural experiment, an off-the-shelf frontier call is the correct choice.
Cost profiles shift dramatically as you climb. Published community findings suggest that while API calls scale linearly with usage, fine-tuning introduces steep upfront compute costs and ongoing dedicated hosting fees. Industry estimates indicate that an open-weight model might eventually lower inference costs, but the break-even point requires sustained, high-volume throughput to justify the dedicated infrastructure.
Before choosing a rung, profile exactly what your task needs: Profile the task, don’t rank the models.