Large language models acquire vast knowledge during pretraining, but post-training techniques like supervised fine-tuning (SFT), reinforcement learning (RL), and online policy distribution (OPD) are crucial for aligning them with human intent. This article systematically compares these methods, focusing on how they shift the model's output distribution from static pretrained patterns to dynamic, task-optimized behaviors. The author explains that SFT anchors the model to specific demonstrations, RL encourages exploration and reward maximization, and OPD introduces a continuous feedback loop that mitigates catastrophic forgetting. For developers and researchers working on model alignment, understanding these distribution shifts is essential for designing effective training pipelines. The piece also touches on practical trade-offs: SFT is stable but limited, RL is powerful but unstable, and OPD offers a middle ground. This analysis is particularly relevant for those building conversational AI, coding assistants, or any application requiring fine-tuned language models.
A deep dive into how SFT, RL, and OPD affect model distributions during post-training, explaining capability learning, generalization, and forgetting.