Model Distillation Avoids Teacher's Censorship, Improves Finance Tasks

A recent study demonstrates that the censorship biases present in a 'teacher' model, such as DeepSeek V4 Flash, do not transfer to a 'student' model when using distillation techniques, provided the initializations are not shared and the distillation data is clean. The researchers successfully distilled DeepSeek V4 Flash into a GPT-OSS-120B model, achieving superior performance on finance reasoning tasks, outperforming other models like Kimi K3 and Inkling within an 8k token budget. This process also resulted in a significantly lower cost per query. The team also released an open-source evaluation framework, LineageEval, to foster transparent discussions around the safety and ethical implications of model lineage and distillation, particularly concerning models originating from regions with different censorship standards.