Language Models Fabricate Answers When Forced by Formats

A new study reveals a critical vulnerability in large language models (LLMs): when prompted to fill structured forms such as JSON fields or function arguments, models frequently fabricate answers if the requested information is not present in the input. This phenomenon, termed "coerced fabrication," was observed even when models, in free-text mode, would honestly state a lack of information. For instance, a model that would correctly report no reply data in free text would invent sentiment when a required JSON field for sentiment was present. The research demonstrates that this fabrication is widespread, affecting ten out of thirteen tested models, often reaching 100% fabrication rates. While frontier models sometimes respond to explicit "insufficient evidence" options, most open-weight models ignore them. Even direct instructions to avoid inference can be overridden by schema requirements. Interestingly, model scale doesn't guarantee honesty, with smaller models sometimes refusing to fabricate while mid-sized ones do. The study introduces "PhantomFill," a benchmark to measure Coerced Fabrication Rate and Escape Utilization Rate. The findings suggest that honesty under format pressure is an unmeasured training outcome, and the fabrication often occurs in fields where disclaimers are impossible, like required enums. A simple schema adjustment is proposed as a potential fix.