Is this the start of something bad?

My boyfriend put his hands on me for the first time last night. To be brief, we both worked that morning. We have a child together so when we got home, he got on the game and I took care of the baby. For 5 hours he played while I fed the baby, played with him, changed him, etc. I asked him one thing, and that was to give the baby a bath. I asked him kindly, then more and more progressively when he didn’t do it. I got upset, unplugged his game and told him to wash the baby. He got on his phone. I told him if he stayed on his phone i’d throw it outside (We live first floor so it would just be in the snow, his phone would be fine.) He laughed said I wouldn’t. I grabbed his phone from his hands, walked to the patio, and chucked it outside. He followed me and told me to get it, I said no. He then proceeded to shove me, hard, up against the wall and I fell to the floor and slammed my shoulder. After this he didn’t say anything, not a sorry nothing. We had family that come over to check on everything. His dad sat us down as I was crying and told him off. Told him that it was not okay and to look at me and apologize. He said that it wasn’t that serious and I was fine, despite my tears. He apologized later crying and then disappeared for 3 hours when I wouldn’t say anything to him. Today, he apologized again, said he didn’t know what came over him. He went to the doctor and got put on some medication and we attended a couples therapy session. He has promised to never do it again and he seems very remorseful. But my question is, is this the start of something bad? Should I leave now? I want to keep our family together, but I do not want to become a victim of abuse. Just need some insigh