Regrets after paying off house?

I've been debating taking some money out of my brokerage account to help pay off my house in a year. About $100k left on the mortgage with only 2.6% interest rate. My brokerage is about $250k. I was thinking of dollar cost averaging and doing like $6k a month so it's not a lump sum coming out. Obviously my rate is really low and the money does better in the market long term. Not sure if the peace of mind is worth not making as much long term. Has anyone done something like this and regretted it?