How the hell do you guys wear boat shoes with no socks?

I decided to class up the slightest bit for the summer, and switched from some size 11 Converse to a pair of size 10.5 Sperrys to wear with khakis and shorts. They look nice, but there's no way I can wear them without socks, unless I'm missing something: they scrape up and down the back of my foot when I walk, which agitates my heel and is probably going to lead to blisters. A size down wouldn't even fit my foot, so that's not an option (my toe barely even fits in 10.5s as it is).

Do people just build up calluses on their heel, or are there some socks out that there that wouldn't show in Sperrys? Or did I just end up with an ill-fitting shoe?