Rules of Contact Question

It seems like if a player has fallen to the floor with the ball, it's not a foul to just jump on top of them and go for the ball, but if you barely touch them while they're dribbling or shooting, that's a foul. Is that the way it's supposed to be called?