Do I let it grow or shave?

So I am about a year and 3 months on T now. And ive grown hair literally everywhere but my face which has started recently. However its just the sideburns and a bit of an uneven neckbeard. I dont like it at all and think it looks musty on me. However I do want to have a moustache and a goatee in the future and with the observable genetics on both side of the family that is very achievable.

Problem is that I've heard conflicting opinions about shaving the face this early. Some told me if I shave like this that I will never be able to grow the beard, while others told me its better to shave. My endocrinologist told me its all in the genetics. I don't know what to think and I know its pretty early but those guys that are on T for a year and can grow full mustaches rlly give me a lot of envy lol.

Id ideally continue shaving, that shouldn't impact my hair growing potential right?