Just bought a horse. Few concerns of the barn owner

I just bought a beautiful quarter horse gelding, I boarded him near my house with a decent family from Spain, I'm paying them $500 per month for Field boarding, I have some issues however which I need your advice on because the last post I posted really blew up lol, let's say I want to visit my horse without his supervision, and it's at the hours of when him and his staff are not at the barn, do you guys think it's ok for me to visit my horse? grab him from the pasture and lead him to the stables and just bond with him, walk around the ring, brush him etc and then eventually let him back to his friends, Sometimes he gives me trouble saying I'm not there, I'm with family and to come in the morning instead, but I have a life of my own as well, with wife and kids, each week I want to visit will be different times and I don't like that he tells me to come 9am or that don't come at 5pm because nobody is there, in the end I still go anyway because I'm paying for this, but I also do understand it's not my barn, it's just my horse on his barn. Let me know what yall think and thanks a lot