How important are the bonus stats?

I just started a few days ago and when I add an adventurer to my list I'v been generally just adding the bonus stats to whatever seem to be their best 2/3 stats (IE A fighter with higher strength and vitality I just split the points between the two). I'v heard from a video that I should dump most of it into speed however and now I'm not sure if I should do that or just do what I've been doing. Does it really matter that much? I'd hate to accidentally screw up an entire characters stats.