Am i screwed if i shift scale?

So ive been really enjoying dungeon blocks but i noticed after printing enough for a few map arrangements that some blocks seem to need some force to slot into place. i could start printing a little under (97/98% scale?) but i worry that'll make all my blocks look mismatched. would a 2-3% change in scale be really noticeable before i waste some filament trying or am i worrying about nothing