Why is this game so difficult?

I just finished the prologue and stepped foot into the open world. I started on duelist difficulty because I figured with Elden Ring and Chivalry experience, the combat wouldn't be too hard, but I've been stuck for hours. I tried clearing the fortress Mert is in after stumbling upon that questline, and I got my shit rocked over and over, so I figured it was like the Tree Sentinel to where I just need to come back later. I've turned the difficulty down, left the area, but no matter what fight I'm in it's just insanely difficult for no reason? I'm in the main paths right outside of the starting area and I have no idea where to even go or who to fight to get stronger.