TRACE Enhances Public Service Chatbots with Trustworthy Retrieval.

Public service chatbots often struggle to provide accurate and reliable recommendations, especially when dealing with noisy and inconsistent service directories and user-specified constraints. General-purpose Large Language Models (LLMs) frequently generate unreliable information, citing unverified sources. This research introduces TRACE, a framework aimed at enhancing the trustworthiness of these conversational systems. TRACE, or Trustworthy Retrieval-Augmented Conversational Engine, focuses on improving retrieval quality for constraint-aware recommendations. It employs a dual data representation schema to effectively parse user queries into both structural and semantic constraints, which then guide the downstream retrieval process. Empirical evaluations using a statewide pantry directory and a synthetic query benchmark demonstrated that significantly strengthening retrieval substantially improves user constraint satisfaction and drastically reduces hallucinated recommendations. A key finding was that as retrieval quality improved, performance differences across various LLMs narrowed, indicating that robust retrieval makes the system less dependent on the specific LLM's size or proprietary nature.