Evaluating Agentic AI Learning Without Labeled Benchmarks

Agentic "Continual Learning Harnesses," which allow Large Language Models to improve from feedback without full retraining, are proving valuable in fields like cybersecurity. However, their evaluation traditionally relies on labeled benchmarks, which are often unavailable, outdated, or unrepresentative in operational security environments. This makes it difficult for practitioners to assess a harness's effectiveness or compare different designs. The proposed framework addresses this by evaluating learning harnesses end-to-end without requiring a labeled benchmark. It leverages the "scaling hypothesis," where a more powerful teacher model provides infrequent, high-precision corrections to a smaller student model equipped with a learning harness. The harness's effectiveness is then measured by how closely the student model converges to the teacher's performance over time. This teacher-relative improvement has been shown to correlate with actual gains against a gold standard, offering a practical evaluation method when labels are scarce.