Paper Argues Against Sole Reliance on Optimization in AI

The paper critically examines the dominant "optimization culture" within artificial intelligence, particularly concerning large language models. It posits that while the engineering advancements leading to more fluent AI outputs are impressive, they stem from a deep-seated belief that measurable improvements along predefined metrics fully encompass the concept of value. This perspective, the authors argue, is a limitation. By tracing this optimization mindset through various stages of the AI stack—from pretraining and decoding to preference tuning, benchmarking, and user interfaces—and back through its historical lineage in the "audit society," the paper identifies a fundamental boundary. An optimization procedure can quantify the improbability of generated text, but it lacks the capacity to discern whether that unlikelihood signifies an error or a genuine act of invention. The authors contend that within a mere half-decade, this procedure, despite its inherent inability to make such a distinction, has paradoxically assumed the authority to dictate what constitutes legitimate language. This authority, historically held by institutions like academies and grammars, has now been transferred to technical apparatuses such as loss functions, reward models, benchmarks, and system prompts, which execute judgment without possessing the actual capacity for it.