How much house can I afford if I make $1M+/year with ~$13M net worth?

I’m looking for some perspective on how much house I can reasonably afford and whether I’m thinking about it the right way.

I’m in my mid 40s, married with 3 kids. My current net worth is approximately $13M, with the following rough breakdown:

• ~78% in equities
• ~6% cash & equivalents
• ~19% in real estate
• No consumer debt

My compensation is highly variable, but generally $1 to $2.5M per year with about $500k in cash and the rest in company shares.

• Annual spending is currently about $180K/year, and we live well below our means.

We’re considering upgrading to a “dream home” in the $3M–$4M range, and I’m trying to sanity check if that’s appropriate given our financial position. I’d plan to buy in cash, using proceeds from the sale of our current home (expecting $700K equity) and sale of securities.

A few things I’m trying to think through:

• I never would have imagined I'd consider buying a home in the couple of million range. I feel like I can afford it, but it's hard to get my head around spending that much.
• I have been considering retiring for the last few years and I feel like I am good financial control -- I worry that I will lose flexibility and control with this much allocated to my residence. I don't want to feel like I'm trapped working ever again.
• I’m not interested in carrying a mortgage at this stage

Would love your views:

• What’s a reasonable home value range in my case?
• Should I be concerned about the opportunity cost of the capital allocation from return bearing to home? It's leaving behind $300 to $400k per year in potential returns.
• What would you consider a "safe" max % of net worth to tie up in a primary residence?
• Any mistakes you’ve seen people make when upsizing in this range?

Appreciate your thoughts.