Fine-tuning large language models with LoRA is a common but often trial-and-error process. This article presents a structured, decision-tree style approach to hyperparameter tuning, based on four rounds of experiments on Qwen3.5-9B. The author systematically varied learning rate, epochs, data volume, gradient clipping, and warmup, tracking final loss, best loss, and token accuracy. Key findings include the impact of learning rate on convergence speed and the role of warmup in stabilizing early training. The decision-tree framework helps practitioners choose the next parameter to adjust based on observed training behavior, reducing guesswork. While the specific numbers are model-dependent, the methodology is transferable to other LLM fine-tuning tasks. This guide is especially useful for teams new to LoRA or those looking to optimize their existing fine-tuning pipelines.
A practical guide to LoRA hyperparameter tuning for Qwen models, based on four rounds of real training experiments. Learn how learning rate, epochs, data volume, and warmup impact loss and token accuracy.