Help with Seas Town

So I just got into the bar, and went to the feather shop after the bandit wanted to battle and I said yes. The problem is, Zan said he'd handle it himself. I didn't go to the battle, I didn't fight the bandit. And every time I go to the feather shop now, the robot tells me that "your partner is waiting inside", and then it shows me Zan staying it's his fight and that he'll handle it. I can't leave town because the others keep asking for more time. It's my first playthrough, please help what do I do.