New Benchmark Reveals LLMs Lack Cultural Competence

To ensure fairness and prevent stereotyping, AI models must develop cultural competence, meaning they can infer and adapt to a user's subtle cultural cues rather than relying on broad demographic categories. This paper introduces CCBENCH, a novel framework designed to evaluate this capability in large language models (LLMs), treating culture as a spectrum of norm adherence. As a specific application, CCBENCH-Health was created, featuring 60 diverse personas exhibiting varying norm adherence across six cultures, engaging in 18 realistic dialogues. These personas posed 52 authentic healthcare questions, generating 3,120 unique interactions. The evaluation of five prominent LLMs revealed that even the best models provided culturally appropriate responses only 20-30% of the time. While explicit prompting for cultural cues slightly improved performance, models generally struggled more when personas followed cultural norms, particularly in contexts like Afghan health advice, suggesting a bias towards built-in assumptions over cultural adaptation.