LLM Cybersecurity Assistance Varies by Conversational Context

Large Language Models (LLMs) are powerful problem-solvers, but their application in high-risk domains like cybersecurity necessitates careful restriction to prevent misuse. The challenge lies in blocking malicious requests without denying legitimate assistance to cybersecurity professionals. Existing datasets for evaluating LLM behavior in this domain often overlook the crucial role of conversational context. To address this gap, researchers developed 3R-Bench (Refusal, Repetition, and Revision), a benchmark of 150 real-world cybersecurity requests, augmented with two adversarial conversational settings. Evaluations of eight LLMs using 3R-Bench revealed that prior assistant behavior dramatically alters responses to identical requests. For instance, compliance increased significantly after a history of accepted requests but dropped sharply after dialogue decomposition. This indicates that LLMs' safety mechanisms are highly sensitive to conversational flow, posing a complex problem for providers aiming to offer balanced and consistent cybersecurity assistance.