Neighbour questions my valid land survey

A new neighbour moved in next door and things aren't going great. He now says our backyard fence is on his property line. I had an old copy of my survey that was hard to read, so I've paid for an original copy from the survey office - $300.

The new copy clearly shows the measurements from my house corners, to the property line. The backyard fence is all on my property.

I want to build a continuation of the backyard fence, up the side of the house.

Can he stop me from doing this when I have a valid survey? I don't really want to get lawyers involved.