Using investments/retirement dollars for primary house purchase is smart, and you should do it.

Obviously earning 8-10% on investments is better than 3-4% on real estate. However, if you're buying a $650,000 brand new house, and you take $130,000 out of your retirement funds to put 20% down on that house, its actually a good thing. Let's take the high end for both investing and real estate (10%/4%). In year one, the house increases in value by $26000, while your investment would only earn $13,000. Eventually, the investment growth will outpace the home, but it takes like 14 years, at which point the house is now worth $1,125,000 and my $140,000 that I took out would have been worth $493694.

Convince me I'm wrong.

Yes, this is coming off a little flippant, and I can think of a few issues with this myself, but I feel like the appreciating asset of a house is sometimes overlooked in conversations around using investment money for a down payment.