ELI5: Why does water from the tap take so long to get hot?

I figure it has something to do with pushing out the cold water that's already in the line, but is it that simple? Does it really take that long to get from the water heater to the tap?