Alpine Loop sizing question

So I got an M and the clasp sits directly at the center under my wrist. If I wanted it to sit up towards one of the sides, could I size up or down - or do I have other size up? I'm worried if I go up there would be a lot more excess fabric looping.