My shiba is at times anxious - should I be worried?

My best friend in the whole wild world, my shiba, is on the sensitive side since he was a puppy. It seemed that he was a shy one around humans from the beginning.

I remember when he was a puppy, he would stand super awkwardly around humans not knowing what to do. I would always take him in my arms whenever I would see him stand around awkwardly. (He very rarely actively wants to be in my arms but is incredibly chill whenever he is in my arms).

He was socialized properly and does NOT stand awkwardly around other dogs. It seems like he knows what to do and he's in his element whenever interacting with other dogs. (70% of dogs, the other 30% he gets into fights with).

I know he loves me but he is not specifically codependent to me, he is codependent to a human presence. I very rarely leave him in the absence of a human presence because I think its just cruel.

He sometimes shows signs of anxiety - to the point of panting, shaking and being a velcro dog to me. For example, whenever we are in a group of 3 people or more who are having fun and being quite loud - the anxiety signs begin.
Recently, I got into a huge fight with my flatmate and until I resolved the conflict, he was shaking. And now, he accepts/is his normal self when he's inside our house but CRIES enormous amounts when I try to get him through the door.

I don't really know what to do about this. I want to give him the best life but I also want to hang out with my pals and have my dog around. I have tried treats, blankets and everything but when he gets anxious, he just gets anxious.

Am I cruel to sometimes just let him be anxious? Is there anything I can do? Can anyone think of any solutions?

He used to be scared of kids (shaking, panting, refusing to move) and after he spent a couple of weeks up close with a 3 year old who loved him but was not gentle at all, he got the gist of what kids are and really likes them now. (The other day, we were at a crossroad and he chose to go down the path where there were 40 kids screaming instead of an empty path).

Same with travelling by train - he is super chill about it although the first train trip was super stressful for him.