how important are the yellow dots?

I can't move on from a camp or castle until I've cleared them all and it's making the game really tedious. I know that what I should do is just move on but my OCD won't let me. I keep thinking, what if I miss some really great piece of loot etc.... What have others done?