Current AI agent evaluations often rely on fixed benchmarks, leading to overfitting and misleading performance metrics. This article proposes a more robust approach: using scenario models to generate new, out-of-sample tasks each evaluation round. Baseline and candidate agents are then compared on these novel tasks, providing a truer measure of generalization. This method reduces the risk of agents 'gaming' the benchmark and encourages development of more adaptable systems. The article also discusses practical implementation considerations for this evaluation framework.
A critique of fixed benchmark evaluations for AI agents and a proposal for dynamic, out-of-sample testing using scenario models.