Which crank length

I am 181cm (5'11") currently with a 86cm (34") inseam andand run 172.5 mm cranks on my road bike, never have had anything else. I am gonna buy new pm cranks, so the time to switch to short 165 cranks would be now. Bike fit wise I have slight lower back issues but I think that could be fixed by adjusting the Saddam height a lil bit.

Does it make a difference, should I Switch?