New Rules Optimize LLM Invocation in Streaming Systems

In streaming inference pipelines, a common challenge is efficiently combining lightweight, fast models with powerful but expensive Large Language Models (LLMs) that offer deep semantic understanding. The critical decision of *when* to invoke an LLM has largely lacked formal treatment. This research frames the problem as a risk-based sequential stopping problem, where an LLM is triggered only when a calculated risk functional, derived from the observation history, surpasses a predefined threshold. The study presents six theoretical results, including guarantees for minimum inter-event times to prevent "chattering," optimality of threshold policies, and sublinear regret for both stationary and changing data streams. It also shows how various classical trigger families can be expressed within this framework. Empirical validation on turbofan degradation data, involving actual LLM calls, confirmed the theoretical assumptions and demonstrated that anomaly-score-driven risk functions significantly outperform other alternatives in terms of cost-efficiency and performance.