Unwanted attention from boss

I work in a corporate team where my boss is the head of department (around 40 people). I’m mid-level and just joined not too long ago.

Recently he’s been giving me quite a bit of one-on-one attention, including asking me to see him in his office multiple times a day or lunch to “discuss work/slides” - lunch is at least 1-2 a week & his treat. I’m a bit uncomfortable because I don’t want office gossip or the appearance of favoritism. Particularly because several other mid-level employees commented that they never had lunch (1 on 1) despite being in the company for years.

He also sent me a late-night message recently saying “just got to my bed,” which didn’t seem work-related, so I didn’t reply. He later on deleted that message.

Separately, I recently made a mistake at work and initially thought it was due to his instructions. I apologised when I realised it was my mistake, and he accepted it without reprimand and even apologised for causing me stress.

Professionally things seem fine, but I’m unsure if I’m overthinking the attention or if I should start setting clearer boundaries (like avoid 1 on 1 personal interactions where possible)

How would you handle this? He is married with children and I am not interested.