Am I unreasonable? Lodger issues

Hi,

I have a lodger who will struggle to find accommodation in this city due to being on benefits. I like him but I'm really struggling and I can't tell anymore if I'm unreasonable/rude or my challenges are something many would struggle with.

Lovely guy, tidy, quiet, friendly.

Never leaves the house as disabled. Literally never.

He lives downstairs with communal areas but it seems that if anything happens downstairs, he makes it his business to join e.g. cooking - he comes to chat or watch, I have friends round - he seems to want to make them his friends and just hangs out with us as long as we are there.

I dont want to be rude but I need some space so I hide upstairs. I know he can hear everything I say downstairs because he sometimes references it later, offers help or laughs from his room.

I live alone besides him so you'd think I'd welcome the friendship but I just feel like he's attached himself unwillingly to my life, like a limpet. He has some mental health problems so I don't want to upset him or kick him out.

Am I unreasonable or is it okay to expect to not feel responsible for his entire social situation/life?