37F and 45M colleagues — is he just friendly and professional, or could there be personal interest?

I’m in my 30s and work in higher education. I have a male colleague in his 40s who works in another department and is not my direct manager.
He is generally known as very formal and reserved with other colleagues. Several people have described him that way. But with me, he seems noticeably more relaxed.

For context, we initially got to know each other through a sensitive workplace matter that he was responsible for handling. It was resolved in my favour and has been completely closed for some time.

He often comes over to say hello first, asks about my health, PhD, work and travel, remembers things I have told him, jokes with me, and our conversations often move beyond work into personal topics like travel, family background, philosophy and life in general.

He also gives me a lot of his time. One meeting that was supposed to last 30 minutes continued for around 45 minutes because we ended up talking about personal interests and travel. On other occasions, even when he has been busy, he has stayed and continued the conversation rather than ending it quickly.

There was also a situation where I became unwell at work and needed medical help. He stayed involved, later gave his personal number to another colleague so that colleague could update him about how I was doing, and when he saw me again he asked about my health more than once.

At one point, I suggested we continue one of our conversations outside work, maybe over coffee. He was very polite but said he preferred to keep things within work and that he generally doesn’t meet people from work socially.

I completely respected that and never asked again. I didn’t ask for his number or keep messaging him.
What confuses me is that after this, he did not become distant at all. He continued approaching me, joking with me, asking questions, remembering details and talking to me in the same warm way.
There is no obvious flirting between us. It is more the difference between how formal he seems with others and how relaxed, talkative and playful he is with me.

I am attracted to him, so I know I could be reading too much into things.
How would you interpret this?
Does it sound like he simply likes and respects me as a colleague but wants firm professional boundaries?
Or could there be some personal interest that he is deliberately not acting on because we work together?

TL;DR: I have a colleague who is usually very formal with others but noticeably warmer and more relaxed with me. We have long conversations about work and personal interests, and I’m attracted to him. I’m trying to understand whether his behaviour suggests personal interest or whether he is simply comfortable and friendly with me.