Beyond Handwritten Prompts: Sampling, A/B Testing, and Automated Evaluation for LLMs

This article challenges the common view of prompt engineering as a one-time writing task. Instead, it presents a disciplined, iterative approach: prompt sampling to explore the model's response distribution, A/B testing to compare prompt variants, and automated evaluation to measure quality at scale. The author explains that good prompts activate the correct distribution in the model, and that over-engineering rules can cause models to follow instructions rigidly rather than enter the right state. For strong agentic models, a lighter touch is often more effective. The framework includes techniques for sampling prompts to discover what the model can do and whether the current path is correct. This data-driven methodology is essential for teams building production LLM applications where prompt quality directly impacts user experience and business outcomes.