I am not riding faster on the main map even though I have a saddle horse for all my troops?

I read that if I have a saddle horse in my inventory for all footsoldier in the party that will mean that my map speed will increase significantly.

That is not the case, even though I have enough saddle horses in my inventory. Currently only at 3.7 :( The rest of the penalties for speed are negligible, no weight, no herding, nothing.