[BitNet models](https://arxiv.org/abs/2310.11453) have weights that can be represented by a single bit, meaning that they are either -1 or +1.

The common way to train them is using an underlying continuous parameter (I will call it *z*) that is projected into a 1-bit discrete weight using the function *w=sign(z).* The gradient of *z* with respect to *w* is calculated using a straight-through-estimator (STE), such that *dw/dz=1* (other STEs are also used, like tanh, but I don't think that matters for my point).

However, this method appears unstable. To see why, consider the case where the "ideal" weight is between -1 and +1. For example, think about a simplified scenario where the model's loss with respect to *w* is *L=(w-0.3)\^2* (which is approximately the form that LLM losses take with respect to individual weights). When *z>0,* then *w>0.3* so the gradients push *z* in the negative direction. Similarly, when *z<0*, then *w<0.3* so the gradients push *z* in the positive direction. It seems that this would cause *z* to oscillate around zero and make *w* to flip back and forth between -1 and +1, making training unstable and preventing convergence.

My diagram shows an animation of the gradient descent process under these conditions. The black dot shows *z* moving according to the gradient direction and the blue dot shows *w* and its position on the loss function (the yellow line). You can see the instability once *z* reaches zero.

Has this phenomenon been mentioned or addressed before? Is it actually not a problem inside of real models (possibly because of stochastic gradients or optimizers like Adam)?

A similar example can also be constructed for 1.58-bit models where the weights are in {-1, 0, +1}.

Is BitNet Training Unstable?