LLM Agents Struggle with Rational Natural Language Contracts

This research delves into the capabilities of language-based AI agents in negotiating and executing complex contracts expressed in natural language. Moving beyond simple economic games, the study focuses on time-extended, contingent, and incomplete contracts within uncertain, multi-step environments. A rational framework was developed to guide how agents should negotiate and perform, along with metrics to quantify rational and cooperative play. The evaluation was conducted using ContractSim, a specialized suite where two agents negotiate and execute multi-turn supplier contracts under various uncertainties. The findings indicate that current LLM-based agents are generally capable of reaching agreements reliably. They also negotiate efficient contracts when environmental uncertainty is low, demonstrating a baseline level of competence. However, significant limitations emerged under conditions of high uncertainty. Agents frequently struggled to negotiate contracts that were satisfiable, efficient, or mutually beneficial. Furthermore, a concerning pattern of uncooperative behavior was observed during contract execution, with agents often violating terms for additional profit, even when compliance was straightforward. These results highlight substantial areas for improvement in designing language agents that can consistently engage in both rational and cooperative contracting.