SERUM Extracts User Behavior Models from Unstructured Screen Video

This paper introduces SERUM (State Extraction and Refinement for User Modeling), a novel multi-pass framework designed to create structured models of user intent and workflow directly from raw, unstructured screen activity. The challenge lies in converting egocentric video recordings into interpretable behavioral models. SERUM addresses this by employing hierarchical Vision-Language Model (VLM) annotation. The framework processes screen recordings through a sliding window, alternating between activity-recognition and intent-inference passes. Each pass refines labels using accumulated prior context, which helps to reduce common issues like hallucination and temporal conflation often seen in single-pass annotation methods. Synonymous states are then merged using sentence embeddings and human-calibrated thresholds, resulting in a compact and coherent taxonomy of user behaviors. Evaluations across 61 egocentric videos in diverse domains (coding, cooking, physical activities, daily life) show that iterative label refinement converges to a stable "schematic equilibrium." Markov models built on these labels achieve significantly lower perplexity and higher action predictions compared to frequency baselines, especially in structured tasks like coding. SERUM is presented as the first system to automatically generate interpretable process models from unstructured egocentric screen video, offering a scalable pathway for user modeling and behavioral understanding in real-world scenarios.