New Framework Optimizes LLM Invocation in Streaming Systems

This paper addresses the critical challenge of efficiently integrating expensive Large Language Models (LLMs) into streaming inference systems. It proposes a novel framework that treats LLM invocation as a risk-based sequential stopping problem. The core idea is to trigger an LLM only when a calculated risk, derived from the observation history, surpasses a predefined threshold. The researchers present six theoretical results, including guarantees against trigger chattering, optimality of threshold policies, and sublinear regret for stationary and non-stationary data streams. This framework unifies several classical trigger mechanisms like SPRT and CUSUM. Empirical validation on turbofan degradation data, involving real LLM calls, confirms the theoretical assumptions. The study demonstrates that anomaly-score-driven risk functions significantly outperform alternative methods, leading to more cost-effective and accurate LLM usage in real-time applications.