Neighbor started replacing shared fence while we were gone, now they want us to split the cost

A little backstory: The fence we share with one of our neighbors has been deteriorating. Over the years, we have replaced a post or repaired a bracket here and there. About a year and a half ago, a section blew down completely and we rebuilt 1/3 to 1/2 of the fence (put in new pressure treated posts, replaced all the rails, and replaced most of the pickets). We purchased all the materials and did the labor ourselves.

A couple of weeks ago, most of the remaining old fence was blown down. We were in the middle of a heat wave, and we were going to be out of town for a week, so we were going to figure out the fence situation when we got back. We just got back the middle of last night to find a letter from our neighbors and the entire shared fence removed (including what we had previously rebuilt) and the posts and rails up for an entirely new fence.

The undated letter from our neighbors detailed their plans to remove all of the existing fence and install a 7 foot privacy fence with overlapping pickets (taller and fancier than the existing style of fence) with work to begin 3 days after we left on our vacation. The letter stated that if we didn't respond by the time the work started (2-3 days notice at best), they would take that as our agreement to share half the $5,000+ cost for replacing (and upgrading) the fence.

I'm honestly speechless at this point. We would have never agreed to remove the portion of the fence we had already rebuilt. We definitely would not have agreed with overlapping pickets with the high lumber prices right now. At most, we would have agreed to replacing the damaged part of the fence with a similar height and style. This new monster of a fence is almost done already.

What can I do to protect myself from paying for half of a fancy fence I didn't want?

Note: This is in California, so normally neighbors are 50/50 responsible for maintaining a shared fence, but there's supposed to be 30 days notice and reasonable costs.

Update: The new fence is almost finished now, and it is such an eyesore. Seven feet looks massive with the size of our backyard, the part where it joins the back fence looks weird because of the 1-1.5 foot height difference, and they decided to have the "ugly" side face our yard for the entire fence instead of alternating like our old fence.