I want to monitor what an install does so I may look for anything malicious. Could some one tell me how to go about this?

I have seen people use these before on youtube, but I can't for the life of me remember the name.

Also, any tips on what I should be looking for?