New Pipeline for Scalable Conversational Agent Evaluation

Evaluating conversational agents, especially in retail, requires more sophisticated methods than simple lexical overlap. It's crucial to assess aspects like intent alignment, factuality, helpfulness, clarity, and tone. While LLM-as-a-judge methods offer a scalable alternative to human evaluation, their production deployment introduces challenges related to governance, reproducibility, cost, schema consistency, traceability, and reliability. To address these issues, the researchers introduce GenAI Evaluation, a governed, configuration-driven pipeline designed for large-scale evaluation of retail conversational systems. This pipeline processes production chatbot logs through a series of steps including normalization, sharding, asynchronous execution, and schema-constrained LLM scoring. It evaluates a comprehensive set of dimensions, including helpfulness, truthfulness, clarity, tone alignment, and translation-specific metrics. The framework incorporates selective re-evaluation, processing only incomplete, malformed, or schema-invalid records, which enhances efficiency. Features like schema locking, versioned configurations, validation logs, and record-level provenance ensure auditability and reproducibility. The pipeline is robust, processing approximately 50,000 records daily and having evaluated over two million interactions. Validation against human-labeled records demonstrated high accuracy, achieving a macro F1 score of 0.93 and 89% human-acceptability accuracy for translation, proving its effectiveness for operationalizing multi-dimensional evaluation.