How to set healthy boundaries

We (me M34 my wife F34 and children) live on an undedicated dead end road. We have many elderly neighbors, one has lived here as long as 75 years.

We are not particularly antisocial, but they are way too comfortable just walking into someone’s backyard to chit-chat. An example is one neighbor down the street whom I have never met and my wife only once, was on our back patio waiting for us to come out to talk. Others will just randomly show up as we are working in the back yard.

I want to be gentle as these are retired people who have lived here a large portion of their lives, but it is also incredibly annoying.

Is there a good way that you have handled a situation like this?