What Am I Supposed To Do?

I play on Xbox and so far my experience has been building up this army of 80 guys, joining as a vassal or a sellsword, getting slaughtered, and being passed around as a prisoner about 5-6 times before I’m back on my feet. Should I level up my fief? Or just keep fighting sea raiders until my guys level up more? I have companions for spotting and training and fighting but it’s still a slow grind.