New AI Framework Detects, Explains Harmful Humor in Memes

Internet memes pose a significant challenge for AI due to their complex blend of visual, textual, and cultural elements, often containing humor, sarcasm, and potentially harmful intent. Existing multimodal AI systems struggle with providing reliable explanations for their interpretations. This new research introduces MAR-12, a framework designed to address these challenges. MAR-12 utilizes Vision Language Models (VLMs) to analyze memes from twelve distinct theoretical perspectives on humor and hate. It then employs a soft-gated attention mechanism to weigh the contribution of each perspective before classifying the meme. The system also generates transparent explanations based on these perspective-specific insights and learned attention weights. Evaluations on datasets like PrideMM and Memotion show MAR-12 outperforming current state-of-the-art methods in both humor and hate detection, achieving up to 80.3% and 75.9% accuracy respectively. Human and GPT-4 assessments confirm the quality of its explanations, especially for memes where humor and harm coexist.