FHSA options with early retirement

Hello, bit of an atypical FHSA question here.

TLDR is my wife and I are about to buy a home and are wondering if we should make qualifying withdrawals using our FHSAs, or use other funds for downpayment and use the FHSAs for additional RRSP room considering we will retire early.

I'm 34, my wife is 37. Soon to have two kids, probably will stop there.

Assets:
1M in HISA (Gift from my parents for house)
~1.15M invested in ETFs/stocks between TFSAs, FHSAs, non-reg, and RRSPs. (This is mainly from inheritance from my grandparents that has grown over 10 years).
No debts aside from the mortgage we're about to take.

Income:
185k HHI (100k me, 85k her)
I have a secure job and work at a crown corporation that I'll stay at until I retire. My income will increase every year and top out at 152k in 13 years, not including yearly COLA increases.

House/Mortgage:
1.5M purchase price, will pay just over 50% down and take a 700k mortgage.

Plan:
I am planning for early retirement at 55, or maybe even terminate earlier, and either take the commuted value of my pension or keep it in the plan. Not sure, but 55 is the latest I will work.

Other considerations:
Because of my pension, I won't accumulate a lot of RRSP contribution room (It's currently at 25k, and only went up 2,300 last year). Given I'm planning to retire early, the extra RRSP contribution room, plus 15 years of growth, could pump that up a lot - the accounts will likely be over 100k each by then. And early retirement will also mean my pension will be reduced by quite a bit. We close in January so we will contribute another 8k each to FHSAs before then, only leaving a year of contributions on the table. Both accounts will have around 40k at that point.

So basically, should I keep the FHSAs open with the plan to convert them into extra RRSP room, or should I withdraw them now and keep that cash invested in non-registered?

I know one of the big benefits of FHSAs are the tax savings at withdrawal, but the lifetime growth in the accounts are about 8,500 each, so the tax (and tax savings) on that if we were to withdraw now won't be that much anyways. It's more a question of if I want more non-registered or RRSP in retirement.

Am I overthinking this and it doesn't really matter? I just want to make sure I'm not making an obvious, stupid mistake one way or the other. Thanks very much in advance.