Early pregnancy and freaking out a bit

Hi there! I have PCOS and was told that I would need to conceive with fertility supports as I don’t get a period or ovulate naturally. I went off BC and was basically waiting for a time that would work for me and my extremely demanding job. I never got a period, but that’s normal. I randomly took a pregnancy test and to my shock, I’m pregnant! I took one of those tests to see how many weeks and it said 2-3, but now I’m about 4-5. I haven’t told anyone but my husband, and I honestly was thinking about not proceeding with the pregnancy as it isn’t an ideal time. But now that time has passed, I’m starting to feel excited about this. My concern is that I have been taking 60 mg short acting Ritalin and haven’t been taking care of myself and drinking coffee and now I’m so scared I damaged my baby. Before I knew I was pregnant I was working stupid long hours and sometimes took a little more than my regular dose and then other days less than my regular dose. I could never live with myself if I inflicted something terrible on another human being! Any advice? I’m freaking out!