When a language model needs to adapt to a specific use case, you have three routes: prompt engineering, context retrieval (RAG), or fine-tuning. These are frequently confused, but each solves a distinct problem. The wrong choice wastes compute and yields poor results. Here is how to draw the line.
Prompt: the cheapest starting point
Prompt engineering means guiding the model with instructions alone, leaving the underlying weights untouched. This is the cheapest and fastest route. No training is required, and iterations take minutes. For many tasks, clear instructions are enough. Start here; only reach for heavier methods when prompting genuinely falls short.
RAG: when the model needs new knowledge
If the model lacks access to specific information β private documents, or data generated after its training run β the answer is usually RAG. This involves retrieving relevant text and placing it in the modelβs context window at inference time. The advantage is that knowledge can be updated continuously without retraining the base model. RAG is the answer to βthe model doesnβt know somethingβ, not βthe model doesnβt know how to do something.β
Fine-tuning: when the model has to change its behaviour
Fine-tuning applies elsewhere: when the problem isnβt missing knowledge, but behaviour. If you need the model to adopt a specific tone, consistently output a strict format, or internalise patterns that are hard to articulate in a system prompt, fine-tuning is the right route. The cost is inherently higher, requiring datasets, compute, and maintenance. Fine-tuning answers βthe model doesnβt do something the way I wantβ, not βthe model doesnβt know something.β
Fine-tuning is itself a ladder
Fine-tuning is not monolithic; it spans a spectrum from light to heavy. The lightest step involves parameter-efficient methods (PEFT) β most notably LoRA and its lower-memory variant, QLoRA. These techniques freeze the original model weights and train only a few small, inserted layers. Compute and memory costs drop sharply, training accelerates, and you can run multiple specialised versions on top of a single base model.
At the far end of the ladder is full fine-tuning, where every weight in the network is updated. It offers maximum flexibility but remains the most demanding option. It requires heavy compute, larger datasets, and strict maintenance, while introducing the risk of βcatastrophic forgettingβ β overwriting the modelβs foundational reasoning abilities.
The core logic remains consistent: if you require fine-tuning, start at the lightest rung. Only consider full fine-tuning when methods like LoRA fail to reach your target quality.
A simple rule for choosing
The decision reduces to a few clear questions. Does a better instruction solve the problem? Prompt. Does the model need information it lacks? RAG. Does the model need to durably change its behaviour or style? Fine-tuning. Often, the right architecture is composite: a fine-tuned model retrieving new knowledge through RAG, guided by a precise prompt.
The cost of each
These routes form an ascending cost ladder. Prompting is the cheapest. RAG sits in the middle, incurring the operational overhead of a retrieval infrastructure. Fine-tuning is the most expensive. Even within fine-tuning, LoRA sits at the lower end and full fine-tuning at the top. Both demand curated data, training cycles, and lifecycle maintenance, but full fine-tuning is far heavier. A common engineering mistake is fine-tuning a model to solve a problem that RAG or a better prompt could have addressed. Before paying the structural cost, verify the requirement.
Where each route hides a cost
The βstart cheap, escalate only when neededβ heuristic is sound, but every tier introduces friction, and honesty requires naming it. Prompting is cheap to write but brittle at scale. A prompt that works today drifts as models or inputs shift, and long instruction stacks quietly consume context windows and compute budgets. RAG provides updatable knowledge but demands a secondary system to build and operate β chunking, indexing, embedding, and adding latency to every call. It also fails in its own way: retrieve the wrong passage, and the model answers confidently from flawed context. LoRA, the recommended fine-tuning rung, is cheap to train but not free to run. You must manage adapter versions, route requests correctly, and accept a quality ceiling below full fine-tuning for profound behavioural shifts. Trading a single general model for a fleet of specialised ones means more assets to evaluate, version, and keep in sync. None of this argues against the ladder; it simply means climbing it with open eyes.
Putting it together
These three routes are not rivals; they are discrete tools for distinct problems. Prompting guides immediate behaviour, RAG provides external knowledge, and fine-tuning internalises durable behaviour. The key to building reliable systems is isolating the exact problem you face β and deploying the cheapest tool that solves it.