New Bandit Algorithm Optimizes LLM Configuration with Budget Constraints.

Evaluating large language model (LLM) configurations presents a significant challenge due to the high costs associated with testing different setups and the need to balance multiple performance metrics. This new research frames the problem as a "cost-aware multi-objective bandit," where each configuration test has a specific cost and yields a vector of noisy outcomes. The study introduces two key algorithms. For online selection, a hypervolume-based UCB algorithm optimizes an efficiency index, achieving a logarithmic budget dependence. For identifying the best configurations under a fixed budget, a cost-aware empirical gap elimination algorithm is developed, showing an exponential decay in error probability with increasing budget. Experimental results on LLM configuration tasks confirm that this framework allows for more efficient decision-making during online evaluation and more accurate identification of optimal configurations, even when resources are limited.