Sooooo...he has a small penis. Help?

I've been dating this guy for a couple months now. He's really great. Sweet, intelligent, respectful, a little shy but lots of fun to be around. We had sex this morning after I stayed over for the night...and it was the worst sex I've ever had. He was 'meh' at oral, came after barely a minute and a half of me blowing him, and then when it got to the main event, he was so small he couldn't even fit inside me during missionary. I got on top and had to thrust extremely shallowly so he wouldn't slip out. He mostly just laid there like a dead fish, sweaty, eyes closed, and completely silent the whole time. After over a half an hour of thrusting, my knees were screaming at me, my toes were sore, and I was covered in sweat. I couldn't even get him to finish before I just rolled off in exhaustion. I apologized that I had left him high and dry and he didn't really give me a reply. Eventually we got up and dressed and after an awkward goodbye (mostly on my part), I left.

God, what do I do? That's officially the worst sex I've ever had. It was far more awkward than it should've been and I feel awful for being upset over it but a good sex life is essential to a good relationship for me and they just can't be separated in my eyes. Am I wrong for being upset about this? I figure I'll give it another go and see if things change but if it's anything like this morning I do not foresee this ending well.

Advice?