Why is Ser Duncan so unpopular/ignored at the end?

Hi everyone, I just watched the whole first series (I know, I'm late lol), but I haven't read the original novellas yet. I don't really get why the other smallfolk are so indifferent toward Duncan in the last episode, after the trial. Raymun's rant against the Targaryens made it seem like most of the ordinary people hated them, so I assumed Duncan would be more popular among the smallfolk, for beating Aerion and causing another Targaryen's death.

Is Raymun just an extremist, or was Baelor more popular than the other Targaryens? Was there more explanation in the novellas that I'm missing out on?