Spacers or no spacer (on a fixie)

I noticed many cool guys are removing all the spacers under the stem (or leaving one small one). Im curious if it is just for the good looks or is there any other reason. I ride mine with 5 6mm spacers and it still feels low... And yes my frame size is correct.... So would you recommend to remove all them spacers coz it actually looks nice with the handlebar down low