Aziz et al. Paper Summaries • 59 implied HN points • 07 Apr 24
- LoRA helps fine-tune large language models without changing all their parameters. It uses two small matrices, which keeps the performance quick during use.
- LoRA's updates to weights can miss valuable details you'd get from full fine-tuning, because it treats magnitude and direction together.
- DoRA improves on LoRA by separating magnitude and direction, leading to better performance on reasoning tasks and other applications. It works best with smaller settings, making it efficient.