Confused about traveling for quests

Noob here, I think I'm missing something when it comes to traveling for quests. I know to follow the "solid" white marker, but it seems like when I arrive at the place I'm brought to by wagon or ship, that I'm still not in the right place. And the person that just brought me there has the white marker again, and takes me back to where I was. What am I missing? What is the best method for getting to the quest location?