Sign in
06 · How LLMs work

Broad intuition · Study map 25 of 85

LoRA: learn a small correction

Suggested preparation: Pretraining, instruction tuning, and distillation

Picture it

Imagine keeping a large machine fixed and adding a small adjustable steering mechanism. LoRA expresses a weight update as two thin matrices. The base remains mostly frozen while those smaller matrices learn.

What to understand

  • Low-rank updates, rank, target layers, merging
  • QLoRA’s quantized base; training memory versus inference memory

Follow the references

  1. 1 · Paper

    Hu et al. · LoRA ↗

    Look at the low-rank update diagram and the rank experiments.

You understand it when…

Why does a tiny adapter file not mean the full model can run in equally tiny memory?

This is a study map. Follow the reference for the full explanation; use the question to test your mental model. Browse the reference shelf.