Sander, smoothing plane, or card scraper?

New-ish woodworker here, trained by YouTube. Everything I’ve done so far has been plywood and I’m getting ready to make the jump to hardwood. Mainly interested in building furniture(I’ll probably start with a basic bookshelf). I own a random orbital sander, a Veritas #4 smoothing plane (sharpened but haven’t really used yet), and a set of blue steel card scrapers that I bought on impulse. I’m also learning how to sharpen.

From what I understand, sanding is the most common way to prepare a piece for finishing. But I’ve also seen videos (e.g., Rob Cosman) who claim that a hand plane can get to the finishing stage much faster, perhaps without even sanding at all. And then there’s the card scraper, which IIRC some say is even better than planing for that final step.

My question is when might I want to use a smoothing plane or scraper instead of, or in combination with, a sander. Also, how would I know when the piece is “smooth enough?” Sanding is familiar to me; I would just step through the grits up to the instructions on the finish I’m using (say, 180 or 220). But with a plane or scraper, how do I know if that’s enough, or if I need to take a final high-grit pass with the sander?