ELI5: How does compass direction inferred from the position of stars allow "celestial navigation"?

I understand that if you know the night sky in your part of the world, the position of the stars appearing on the horizon allows you to tell where north is. Obviously that's a hugely important part of navigation. But even if you had a compass, how does that help you find a small island in the Pacific Ocean, for instance, without a good way to tell distance travelled? If you're a navigator crossing the Atlantic, and the main objective is to continue approximately south west, I see how that would help you, but as a Polynesian seafarer, your targets a pretty damn small and I don't see how even perfect compass direction would enable you to sail from island to island given that over time, you'll drift a little, even if your heading stays true.