CallBench Evaluates Dual-Goal Coordination in Phone Assistants.

While target-oriented dialogue systems have become adept at fulfilling explicit user goals, phone call assistants face a more complex challenge: coordinating two distinct goals simultaneously. These systems must balance the device owner's pre-set explicit goal with the caller's often implicit and dynamic objective. To address this, researchers have introduced CallBench, a new Chinese benchmark specifically for evaluating dual-goal coordination in phone call assistants. This comprehensive dataset includes 50,000 complete multi-turn dialogues spanning six common scenarios like takeout, delivery, and taxi services. CallBench covers various goal relationships, including alignment, complementarity, irrelevance, and conflict, and incorporates a preset-aware, turn-level evaluation protocol. Initial experiments with existing dialogue methods demonstrate that current approaches are not yet proficient in this dual-goal task, highlighting a significant need for more advanced phone call assistants capable of making reliable, turn-level decisions under proxy constraints.