ERGO Optimizes Prompts for Text Classification via Error Analysis

Prompt optimization for text classification encompasses various approaches, including demonstration selection, exploration-based search, and error-driven diagnosis. Each method has distinct strengths and weaknesses, and no single technique consistently outperforms others across all task types. A comprehensive empirical study revealed that different paradigms excel on structurally different classification tasks. Guided by these insights, the researchers propose Error-Guided Optimization (ERGO). This error-driven method operates by iterating over the full training set in non-overlapping batches. It systematically diagnoses classification failures and then generates targeted decision rules through a "diagnose-prescribe-rewrite" feedback loop. ERGO demonstrates superior accuracy on tasks characterized by errors concentrated in specific confused label pairs, termed "boundary-learnable tasks." For instance, it achieved 90.0% on TREC and 94.4% on CLINC150, converging efficiently within 3-5 iterations. A key advantage of ERGO is its ability to produce interpretable decision rules. While it may not achieve the highest overall average across all benchmarks, ERGO fills a complementary role: demonstration-based In-Context Learning (ICL) performs best on coverage-dependent tasks, exploration-based search on many-class intent, and ERGO shines where decision boundaries can be learned from error patterns. The paper provides a framework linking task characteristics to the optimal prompt optimization paradigm.