On-Device AI Research Agents: Faithfulness and Coverage Factors

This research explores the performance of on-device AI research agents, specifically focusing on the faithfulness of their citations and the trustworthy coverage of their generated briefs. Using a 4B parameter model running on a standard laptop, the study disentangles two key metrics often conflated in evaluations. The findings indicate that the amount of source material an agent is exposed to directly impacts citation faithfulness. Increasing the character count of each source provided to the generator significantly improved the accuracy of cited claims, regardless of whether the sources were "gold" standard or retrieved. This suggests faithfulness is primarily bound by exposure. Conversely, the study found that trustworthy coverage – whether the agent cites the *right* sources – is predominantly determined by the quality of the initial retrieval process. Exposure had little effect on coverage, as recall remained consistent. The practical implication is to first optimize source exposure for faithfulness, then focus on improving retrieval recall for comprehensive coverage.