Thermodynamics question about heating your house.

There has been some debate about whether it is more efficient to turn your heat off when you leave the house or to leave it on/turn it down. I've heard that if you turn the heat off when you leave, the furnace has to run much longer when you get back to catch up and reach comfy zone. It seems true because it takes a lot of energy to heat up a house. The building has a heat capacity that depends on how well insulated it is. And if you keep the heat on all day at say 20C some heat will radiate back inside the house from the walls/insulation thus requiring less energy input over time. Is this a myth?

Others say that keeping the heat on all day wastes heat, because you aren't there to use it.

Alright scientists: GO!