LoRA is the baseline, but it is not the only option. Since its introduction, several variants have emerged to target specific constraints: memory, quality, or speed. Understanding these trade-offs helps you choose the right approach for a given workload.

QLoRA: when memory is tight

QLoRA is designed to save memory. It keeps the base model compressed at 4-bit precision (using the NF4 format), restoring it to higher precision only during computation. The impact is significant: a 7-billion-parameter model requires roughly 6 gigabytes of memory instead of about sixteen. Crucially, quality remains largely untouched, performing very closely to standard LoRA. If GPU VRAM is limited or you need to fine-tune a larger model on constrained hardware, QLoRA is the default choice.

DoRA: when quality comes first

DoRA addresses a different problem: closing the quality gap between LoRA and full fine-tuning. It decomposes each weight into two components: magnitude and direction. DoRA manages these separately β€” applying LoRA to the direction while learning the magnitude independently. This separation improves quality by a few percent, particularly on complex tasks like reasoning and code generation, with under 5 percent overhead on the parameter count. If quality is critical and standard LoRA falls short, DoRA is the logical next step.

LoRA+: when speed matters

LoRA+ is a subtle but effective structural adjustment. In standard LoRA, matrix B initialises at zero and must travel further to reach its optimal value. LoRA+ compensates by assigning B a larger learning rate than A. The result is faster convergence β€” reaching target quality roughly 1.5 to 2 times faster. If the training budget is strict or rapid iteration is necessary, LoRA+ saves time with only a minor configuration change.

A simple decision tree

The choice reduces to a few operational questions:

  • Is the primary constraint memory? β†’ QLoRA.
  • Is quality critical and standard LoRA shows a gap? β†’ DoRA.
  • Is training speed the highest priority? β†’ LoRA+.
  • None of the above? β†’ standard LoRA, which remains highly effective for most workloads.

Putting it together

These variants are not rivals; they are specialised tools for specific constraints. They are also not mutually exclusive β€” their underlying mechanisms can be combined. But before adding complexity, start with the baseline: try standard LoRA, and only when you hit the limit of a specific constraint, reach for the variant that targets exactly that constraint.