Falling to the side for crow pose

I’m currently practicing the crow pose (including baby crow), and while there were times that I fell forward, a lot of the times I’ll lose my balance after a second or two and fell to the side instead (and it is always the right side). Any tips or reasons as of why it happens and what I can do to improve it? Thanks :)