CRAFT Diagnoses LLM Weaknesses for Targeted Fine-Tuning

Current LLM evaluation pipelines often identify *where* a model fails (e.g., on certain examples or topics) but not *why* it fails, leaving the underlying capability gap implicit. This makes it difficult to generate effective data for model improvement. The CRAFT method addresses this by providing a model-specific diagnosis of weak capabilities. CRAFT transforms any rubric-based evaluation dataset into a detailed capability assessment. It treats each grading criterion as a probe, extracting capability descriptions from prompt-rubric pairs. These descriptions are then clustered into a hierarchical capability tree. The target model is scored at each node of this tree, allowing for dynamic selection of low-performing nodes at the granularity where the failure is most evident. The identified weak capabilities directly inform the generation of targeted supervised fine-tuning data. When compared against other clustering methods and untargeted data generation, CRAFT consistently produced stronger models. For instance, it achieved the best average performance for all four tested models in the finance domain and for three out of four in the legal domain, demonstrating that diagnosing weaknesses at the rubric criterion level leads to measurably better models after fine-tuning.