Bought some loafers but they are hurting me

Hi MFA!

I recently bought these loafers.

They seem to fit well but after some walking around it seems that the side of the shoe bangs against the bottom of my outer ankle bone. After enough walking it seriously starts to hurt.

Not sure if it's just that the shoes need breaking in or what. Any advice? New insoles perhaps?

Thanks.