Fine-tuning with LoRA exposes a small surface area of hyperparameters, but getting them wrong often separates a useful model from gibberish. Once you understand the mechanical purpose of each, setting them stops being guesswork.

Rank: the size of what you learn

The rank (r) determines the dimensions of the low-rank matrices LoRA injects, which defines the model’s learning capacity. A larger rank absorbs more complex patterns but increases computational cost and the risk of overfitting. A practical baseline by task:

  • Style and formatting: r=4 to 8
  • Instruction following: r=8 to 16
  • Domain knowledge: r=16 to 32
  • Complex tasks: r=32 to 64

A sensible default is r=16. If the model underfits (training loss plateaus too high), increase the rank; if it overfits (validation loss diverges), decrease it.

Alpha and the alpha-over-rank scale

Alpha (lora_alpha) has no standalone meaning; what matters is the ratio Ξ±/r, because LoRA’s contribution to the output scales by exactly this fraction. The standard practice is to set Ξ± = 2r (yielding Ξ±/r = 2). This ensures that if you adjust the rank later, LoRA’s relative contribution remains constant. If training grows unstable, Ξ± = r offers a more conservative scaling, and if you set Ξ± significantly higher, you must compensate by lowering the learning rate.

Learning rate

LoRA tolerates higher learning rates than full fine-tuning because it updates far fewer parameters. A standard functional range is 1e-4 to 3e-4. Starting at 2e-4 with a cosine decay schedule and roughly a three percent warmup is reliable. If your model outputs gibberish post-training, check this first; an excessive learning rate is the primary cause of catastrophic degradation.

Dropout

The dropout parameter mitigates overfitting. The optimal value scales inversely with dataset size:

  • Fewer than a thousand samples: 0.1
  • Between one and twenty thousand samples: 0.05
  • More than twenty thousand samples: 0.0 to 0.05

Sparse datasets carry a higher risk of memorisation, requiring more aggressive dropout to force the network to generalise.

Which layers: target_modules

Modern practice defaults to "all-linear", applying LoRA to every linear layer in the architecture. Older conventions that targeted only attention projections (q_proj and v_proj) have largely been deprecated in favour of the quality gains from comprehensive targeting. Attention layers handle routing, but MLP layers store factual associations; if a task requires net-new domain knowledge, both must be updated.

Number of epochs

Epoch count should run inverse to dataset volume. Smaller datasets require more passes:

  • Fewer than a thousand samples: 3 to 5 epochs
  • One to ten thousand samples: 2 to 3 epochs
  • Ten to fifty thousand samples: 1 to 2 epochs
  • More than fifty thousand samples: 1 epoch

Over-iterating on large datasets reliably induces overfitting. A single epoch is frequently sufficient for production volume.

Putting it together: a safe starting point

If you need a baseline, this configuration handles most initial runs safely: r=16, Ξ±=32, a learning rate of 2e-4, target_modules="all-linear", dropout scaled to your data, and one to two epochs. Monitor the loss curves: converging training and validation loss means the configuration is sound. If validation climbs while training drops, reduce the rank or the epoch count. These few numbers decide ninety percent of the result.