If astronomical changes take millions and billions of years to occur, than how do we have as much solid scientific evidence for these types of things as we do?

For example, how can we say for sure that stars age the way they do when any kind of observations of it happening are impossible at the moment? How do we know the universe is expanding?

Is all of it based off of inferences that we make from other things we see in the universe? Or is the science of astronomy really not as solid as I thought?