Down payment for a secondary rental property- TFSA or taxable account

If your tfsa's are maxed out, are you better off saving up money in a taxable account for a down payment on a home over a few years or using money in the tfsa for the down payment and then slowly replenishing it over a couple of years. Likely about 100k withdrawn. There is no rush to purchase the property, I'm just wondering what is a better strategy. The trade-off appears to be between paying taxes or losing compounding interest over the duration of replenishment.