LLMs Offer Inconsistent Parenting Advice, Language Influences Responses

As more individuals turn to large language models (LLMs) for advice, including sensitive topics like parenting, the need for robust evaluation beyond simple information quality becomes critical. Researchers developed a human-centered benchmark, utilizing a multi-dimensional rubric created by parenting experts, to assess 15 LLMs across 100 parenting scenarios in both English and Chinese. The evaluation employed an LLM-as-a-judge methodology. The findings indicate that while aggregate scores might appear reasonable, they often mask specific weaknesses within the LLMs' advice. Critically, different models implicitly promote varying parenting styles, and the language used for the query significantly impacts the nature of the responses. The study underscores the complexities of evaluating LLM-generated advice in socially sensitive domains, emphasizing the importance of auditability for evaluation outputs. This research provides crucial insights for developers and users considering LLMs for direct user engagement in advice-giving applications.