Downsizing to CRV

Has anyone downgraded from a Highlander to a CRV? We are a family of 4. I have a 2 year old and a 4 year old. Im looking into the 24 CRV Touring sport Hybrid. Im just looking to save on gas anf car payment. Thinking I dont really need such a big vehicle but I am so torn! I love my Highlander but we are trying to save money!