Fine-tuning can feel opaque when it fails. Fortunately, most failures trace back to a few recurring patterns, and each leaves a clear signature. Here are the most common pitfalls and how to spot them.
The loss wonβt come down
If the loss stays roughly flat during training, you have three primary suspects: the learning rate is too low, LoRA is not applied to the correct layers, or the Ξ±/r ratio is too small. First, check the number of trainable parameters. If that number is unexpectedly small, your configuration is likely incorrect. If the parameter count is expected, incrementally raise the learning rate.
Overfitting
If the training loss decreases while the validation loss climbs, the model is memorising the data rather than learning the underlying pattern. To fix this, reduce the number of epochs, raise the dropout rate, lower the rank, or implement early stopping. Overfitting is particularly common on small datasets.
Gibberish output
If the model produces garbled or meaningless output after training, the cause is usually one of the following: the learning rate is too high, Ξ± is too large, you have trained for too many epochs, or the training data is corrupted. Lower the learning rate first; this is the most common culprit for broken output.
Running out of memory
If you run out of memory during training, apply these mitigations in order: enable gradient checkpointing (this yields the largest saving), drop the batch size to one and compensate with gradient accumulation, use four-bit mode (QLoRA), and finally, shorten the sequence length. A combination of the first two is usually enough to resolve the issue.
Slow training
If training is unexpectedly slow, you have a few structural options: use Flash Attention on compatible hardware, use bf16 instead of fp16, and increase the batch size if memory allows. The bottleneck is often found in suboptimal settings rather than the hardware itself.
Data quality
Sometimes the metrics look correct, but the output remains poor. In this scenario, suspect the data. Manually review a sample of the training records. Check whether the formatting is exact and if the examples are high-quality and consistent. A small, clean dataset almost always outperforms a large, messy one.
The evaluation gap
The final pitfall is failing to observe the problem at all. Never rely solely on training loss. Always measure performance against a separate validation set, and qualitatively compare the output against the base model. Without this baseline comparison, you risk believing you have improved the model when you have actually degraded its quality.
From pitfall to checklist
These pitfalls have clear signatures, meaning debugging does not have to be guesswork. When a fine-tuning run fails, read the symptoms β flat loss, overfitting, or gibberish output β and investigate the specific suspects linked to them. Running through these checks resolves almost all early training problems.