Frontier LLMs Show Collective Blind Spot in Oncology Decision-Making

While Large Language Models (LLMs) excel at medical knowledge tests, real-world oncology involves complex decision-making under uncertainty, not just factual recall. Existing benchmarks don't fully capture this. Researchers developed the Oncology Decision Boundary Benchmark (ODBB), comprising 2,005 oncology decision points based on NCCN guidelines and colorectal cancer cases, to evaluate nine frontier LLMs. The study found a significant collective failure: 42.1% of all items were answered incorrectly by all nine models. This failure rate was higher for case-specific scenarios (66.4%) than for NCCN guideline items (35.7%). The errors were concentrated in the meta-judgment of choosing between guideline pathways, rather than reasoning within a chosen path, indicating a consistent "blind spot" that architectural changes, not just more data, might be needed to fix. Furthermore, models tuned for decisiveness made unsafe commitments more frequently without achieving higher accuracy. A notable finding was that models sometimes identified the correct next step but failed to commit to it, highlighting a decision-making rather than knowledge gap. The research concludes that model quality isn't the primary bottleneck for clinical LLM deployment; rather, the assumption that a single model can be the sole basis for a clinical decision is the limiting factor, suggesting a need for architectures that detect competence boundaries and route decisions to clinicians.