Authors release the QATFactory tool and checkpoints for training LLMs with target quantization

The authors have released the complete training code for the QATFactory tool and the resulting checkpoints. The open-source tool supports quantization-aware distillation (QAD) and reinforcement learning adapted for quantization (QARL), the NVFP4, MXFP4 and Q4_K formats, both dense and mixture-of-experts models, and training of all parameters as well as via LoRA.

During training, the tool simulates the quantization used at deployment, while performing matrix computations in BF16. According to the authors, this enables training for NVFP4 on H100 GPUs, which do not have FP4 Tensor Cores. According to the authors, checkpoints can be exported directly into vLLM and llama.cpp without any further lossy conversion and without additional overhead during inference.

The experiments covered models with 8 to 230 billion parameters. For the Qwen3.5-9B model, the QAD method achieved an average benchmark accuracy of 68.9% for NVFP4 and 66.0% for MXFP4; the best post-training quantization (PTQ) results were 65.4% and 56.4%. According to the authors, the appropriate strategy varies by format: for NVFP4, quantizing only the weights during training generally performed better, while for MXFP4, quantizing both weights and activations performed better. With the same number of training tokens, using a smaller number of 32K-length sequences increased average accuracy by 1.9 points compared to a larger number of 4K-length sequences.