Land dispute mass

Location: Central Massachusetts

We were sold a home on a lot with a fence line. We were told part had just been replaced but the back would still need to be done.

I tore out the yard plants and shrubs and built garden boxes along the fence and replaced with white picket fence so more sun came through.

Neighbors are now claiming 3 years later that we actually are on their land based on markers listed on their deed.

The online map for the town shows they might actually be correct and I've actually been encroaching on their land and that the fence and bushes might not have been mine to remove and replace 3 years ago?

They've said it's fine as it's just a garden, but now a surveyer has showed up. I'm worried that this could incur a legal process? I honestly don't care, I can move the garden, I can retreat back into my land, unfortunately I can't do much about the fence I replaced thinking it was mine nor the shrubs that were removed, outside of replacing them.

However financially we are pretty tight. With the cost of oil this winter we are still financially recovering.

What should I be prepared for?