[KCD2] Game seems rigged?

Not entirely sure how you articulate this but:
Finished the opening, thoroughly enjoyed, world finally opened up
And it seems every direction I walk to I’m getting killed?
Played some dice and won 3 coin, stumbled on bandits (?), kill the first but die to the second
Go in a different direction, run into 2 wild dogs and I manage to kill one and the other gets me
Perhaps I need to stay in town and do side stuff or something but I do want to get out there and fight stuff and explore
The 3 deaths basically erased 40 minutes of playtime so I was back to where you meet the beggar
Am I stupid? Surely there’s a way to play this game just for the story