ReGraph Generates Structured Recipe Graphs from Food Images

Recent advancements in Large Multimodal Models (LMMs) have shown impressive capabilities in generating recipes from food images. However, cooking is inherently a structured, transformational process where ingredients undergo specific state changes through ordered actions. The free-form text recipes generated by LMMs often leave these critical procedural details—such as entities, intermediate states, and dependencies—implicit and entangled, making it difficult to assess if models truly understand the underlying cooking process. To address this limitation, researchers have introduced ReGraph, a comprehensive, large-scale dataset of recipe graphs. ReGraph explicitly represents cooking entities (ingredients, actions, tools), describes ingredient state changes using attributes, and encodes manipulation targets, destinations, and procedural ordering through typed relations. It also includes explicit Recipe Reasoning Chain-of-Thought (RR-CoT) traces, providing additional supervision for decomposing procedures and generating structured graphs. Building on this dataset, the paper proposes Recipe Graph Learning (RGL), a two-stage framework that enables LMMs to generate a plausible, fine-grained cooking workflow as a structured recipe graph from a food image. Experiments using a deterministic, schema-aware matching protocol reveal a substantial disparity: while existing LMMs produce competitive text-generation scores, they yield limited reference-aligned entity and relation structures under the ReGraph schema. In contrast, RGL consistently improves the generation of cooking entities and procedural relations across different LMM backbones, though capturing fine-grained ingredient states remains the most challenging aspect.