England - Giving permission for access across garden (vs 'a right')

My neighbour currently crosses the end of my garden with their ride-on mower to access their parents’ property on the other side of me (the fence has removable panels on my side).

The farm beyond our gardens has a right to vehicular access across this strip of land (hence the removable panels), but there is nothing in any of the relevant property deeds giving my neighbours such a right. They have said “your predecessors were kind enough to give us the right of access”. I asked for a copy of relevant documentation, none was produced (I do not believe it exists).

My understanding is that, unless it’s recorded in the deeds, I am not bound by any ‘right’ granted by my predecessor. He just let them cross for the year or so before I moved in, I think. In other words, he gave them permission, rather than ‘the right’.

Whilst I don’t intend to be awkward, I do wish to protect my property. I am happy to give my permission for them to use this route once a week for mowing purposes, but nothing else. I believe that if they continue to use this access route without my permission, then they will eventually gain historic access rights, which I do not want to happen.

Is giving them express, specific permission the best way to approach this situation? Making it clear that I believe they do not have a right to access as no evidence of this has been produced, that anything outside the permission I’ve granted will be considered trespass, and I reserve the right to withdraw permission at any time. Would this be a sensible and reasonable approach?

Thank you.