Wayfarer Leveling too Fast

I'm using Perkus Maximus but because I never fast travel I'm finding that wayfarer is leveling up way too fast. Is there any way to drastically decrease the rate at which wayfarer levels?