The problem for this comparison is that QAT always requires doing some extra training iterations. So even if we train the bf16 checkpoint for longer, the QAT checkpoint one will again need extra compute. You can always do QAT on an earlier checkpoint, to try to simulate the same amount of compute. However, this is not a real-world scenario, because we cannot chose between deploying a bf16 model trained for longer or a 4-bit model trained less time, as the bf16 doesn't fit in the target deployment hardware.
In any case, take into account, that while the bf16 compressed model was trained in millions of samples, QAT only requires an amount of training samples in the thousands. So it is not like we are doing a huge training for QAT. The data used for QAT is also a subset of the data used for compression, so no new knowledge is intruded in this phase.
As I understand, your main criticism is around the claim that the 60B 4 bit model outperforms the 60B BF16 checkpoint, and that this wouldn't happen if we had trained the BF16 checkpoint for longer. This might be true, also, GPT-OSS is an special case, as the original 120B model was already a 4-bit model. Theoretically, if you would plateau the 60B model until it cannot learn anything because all the parameters of the model are exhusted, it would never be posible to do a lossless quantization. So I think that you are right here, in our experimental setup with our data and hardware constrains we got a 4-bit model that outperformed the bf16 one, but this is not always expected. But what we wanted to share is not "a quantization method that makes a 4-bit model better than a bf16 one", but the recipe that worked for us to minimize the loss of precision during a compression and quantization process. In this particular case, this loss was so minimal, that in fact, we outperformed the bf16 model, but this is not what you should expect from every quantization. But you should totally expect that:
- If you use the original model as teacher, instead of the compressed one, you will get better performance.
- If you use KL divergence instead of cross-entropy loss, you will get better performance.
There are some ablations left on the table that we would have loved to perform, for me the most important one, is to perform QAT with the 60B bf16 model as teacher instead of the 120B one. But the fact that the 60B 4-bit model outperforms the 60B bf16 model, is a very strong indication, that using the 60B bf16 model as teacher would perform worse, as the performance ceiling would be lower. I am the first one that would have loved to run more experiments, unfortunately, the GPUs were needed elsewhere and we had to move on. We wrote the report with the data we had, and we hope that some people will find it interesting.