Research compares methods for preserving model safety during repeated fine-tuning
A study on the Mistral-7B and Llama-3-8B models compared two approaches to maintaining safety alignment during sequential LoRA fine-tuning: supervised replay and logit distillation. Distillation achieves a smaller reduction in attacks, but damages the model's willingness to answer harmless queries far less.
A research team examined how safety alignment of the language models Mistral-7B and Llama-3-8B degrades during sequential LoRA fine-tuning (progressive fine-tuning on code generation, German summarization, and scientific questions), and compared two remediation methods: supervised replay (repeatedly inserting refusal and compliant examples) and logit distillation (adapting output logits to the base model without an additional supervision loss).
According to the study, supervised replay achieved the largest reduction in attack success rate on the HarmBench benchmark (ASR), on average 51.1 points below the baseline level for the Mistral model, but at the cost of a significant increase in unnecessary refusal of harmless queries – on average 53.4 points above the baseline level on the OR-Bench-hard benchmark. Logit distillation without a supervision loss achieved a smaller reduction in ASR (5.4 points), but increased over-refusal by only 5.1 points. The Llama model showed a similar asymmetry between the two methods. According to the authors, combining both approaches mitigates the excessive refusal typical of replay without significantly affecting performance on downstream tasks.
Geometric analysis revealed a difference between the models: for the Mistral model, distillation better preserves the refusal direction derived from the base model than replay, whereas for the Llama model, refusal behavior after replay persists even after removing (ablating) this direction. The authors conclude from this that preserving the base refusal direction is not a reliable indicator of a model's actual behavioral safety.
The work is published as a preprint on arXiv (arXiv:2610.38001v1). See the source article for details.
Why it matters
The results are relevant for teams that repeatedly fine-tune open-weight models on company data – they show that aggressive safety replay can unnecessarily block legitimate queries, while gentler distillation maintains safety at a lower side cost. The findings also warn against relying on simple geometric metrics (refusal direction) as the sole test of the safety of a deployed model.
Relevant practical impact
What this means
For a business
Companies performing sequential fine-tuning of open-weight models (e.g., for progressively adding domains or languages) can, by choosing a combination of logit distillation and supervised replay, reduce the risk of over-blocking legitimate queries compared to a pure replay approach, while maintaining comparable security resilience against attacks.
DevelopmentCheck the original
Event sources
only one source so far · 1 publisher, 1 independent. We count feeds from the same owner only once.