SCARCE Estimates Rare AI System Failure Probabilities

Estimating the probabilities of rare events, which are critical for the safety and reliability of modern AI systems, is notoriously difficult due to the prohibitive sample budgets required by direct Monte Carlo simulations. Subset Simulation (SS) offers a solution by breaking down rare-event probabilities into more manageable conditional probabilities over nested intermediate events. However, classical SS demands a manually designed scalar performance function, requiring detailed knowledge of the failure mechanism and limiting its applicability to new domains. SCARCE (Scalable Cascade Analysis for Rare-event Characterisation via Embeddings) overcomes these limitations by replacing the performance function with learned latent representations and geometric rulers. These rulers score the proximity to failure regions, while adaptive thresholding directly constructs nested intermediate events from data. The method is formalized with a non-negative supermartingale, providing a valid upper envelope even with early stopping. In experiments, SCARCE demonstrated superior accuracy, achieving 400-500 times lower mean absolute error than traditional SS on MNIST misclassification tasks. When applied to LLM jailbreaks under a fleet-level threat model, a PCA-based ruler within SCARCE achieved low mean relative error on Llama-Guard-3-8B hidden states and successfully transferred to a GCG-style corpus, proving its effectiveness in characterizing rare AI system failures.