REI rainier pants - large too big

Heading out to Machu Picchu next week and preparing all my gear. Got REI rainier pants today and large seems to big on top of my OR Ferrosi pants. I am 5’11”, 182 lbs and 36” waist.

I will try and see if my local store has M in stock as I don’t have enough time to get it shipped

Any inputs on size or a different brand to try would be great.