CodeBlock Improves LLM Code Generation with Structure-Aware Supervision

Traditional supervised fine-tuning for code-generating Large Language Models often treats all tokens equally, applying uniform loss. This approach can be inefficient and may not capture the structural nuances of code. While some methods select high-value tokens in natural language, directly applying this to code can disrupt its syntactic and semantic integrity. CodeBlock addresses this by introducing a structure-aware sparse supervision framework. It first identifies high-quality instruction-response pairs, then breaks down code responses into coherent programming units. These units are evaluated for their utility based on core logic tokens and then re-ranked considering data-flow dependencies. During training, the full code response is available as context, but the loss is only applied to these carefully selected, structure-complete code items and informative natural language tokens. This method achieves superior code generation performance on benchmarks while using a significantly smaller fraction of supervised tokens, demonstrating more efficient and effective learning.