Abstract
As Large Language Models (LLMs) continue to evolve, practitioners face increasing options for enhancing inference-time performance without model retraining, including budget tuning and multi-step techniques like self-reflection. While these methods improve output quality, they create complex trade-offs among accuracy, cost, and latency that remain poorly understood across different domains. This paper systematically compares self-reflection and budget tuning across mathematical reasoning, text-to-SQL generation, sentiment classification, and translation tasks. We evaluate prominent LLMs from the Amazon Nova and Anthropic Claude families under varying reflection depths and compute budgets to derive Pareto-optimal performance frontiers. Our analysis reveals substantial domain-dependent variation in self-reflection effectiveness – with performance gains up to 220% in mathematical reasoning but mixed or negative effects in translation and SQL tasks. We further investigate how reflection round depth and feedback mechanism quality influence performance across model families. Additionally, our findings were validated through a real-world case study at Lounge by Zalando, where self-reflection showed market-dependent effectiveness, reinforcing the importance of domain-specific evaluation when deploying these techniques. Our results provide actionable guidance for selecting optimal inference strategies given specific domains and resource constraints.
Cite
@inproceedings{butler2025finding,
title={Finding the Sweet Spot: Trading Quality, Cost, and Speed During Inference-Time {LLM} Reflection},
author={Butler, J. and Kozodoi, N. and Afolabi, Z. and Tyacke, B. and Baimuratov, G.},
booktitle={{KDD} 2025 Agentic & GenAI Evaluation Workshop},
year={2025}
}
