Playing with/adjusting a very scared and nocturnal rescue cat

Hi all, not too sure where to begin with this and don't know if I'm doing something wrong honestly! I got a rescue cat about 6 weeks ago. He is roughly 4, was on the street for 2 years then been in a few foster homes. I was aware he was very shy and takes time to settle. He's indoor only due to health (FIV) but no problems.

Initially he was hiding all the time, over a few weeks he got a lot more confident. In the day, if I lie on the bed he will now often let me pet him and come over to lie on me, loves being pet, butt scratches, just being lazy lol. His routine roughly - he sleeps in a corner next to my bed (he has a dome bed he can hide in, mostly just lies in front on his pillows/blanket). On days I work I leave about 6 - he normally gets up when I go, or around that time if I'm staying home anyway. Goes downstairs, run around, eat etc. Then spends most the day in his bed or my bed, on windowsill maybe.

He gets the zoomies in the evening, too. A lot he waits until I get in bed, maybe 5-10 minutes he is up and downstairs, comes up and down throughout the night.

I've got some toys for him, a cat tunnel, some boxes, I know he loves it all but problem is I can't really play with him!

He is fine me petting him in bed but when he's exploring and we are around, he hides. I use a string toy in my room but he hides under a chair. He'll play with it if it is close to him but ONLY when he is somewhat hidden - not in the middle of the room.

If I'm going up/down stairs and he is going the other way - he runs back, or hisses if I move closer. I have to step aside (like retreat into the bathroom) to give him space to pass..

As he spends the day in bed and doesn't want to play then, and is running around at night when I'm asleep, I just don't know what to do.

Do I try playing with him during the day when he's in bed? If I go downstairs when he is there he just hides under the table. Sometimes a treat helps him come out but he goes back quickly. Do I keep following him when he gets up so he is used to me, or leave him be and keep giving him time?

Sorry for all the rambling. I just want him to be happy. He is very lovely and can be full of energy.

I don't mind him being up at night, but I'm worried he needs more human stimulation and play time. And as he doesn't seem to respond much to my attempts most the suggestions I'm seeing don't help much.

Thank you for any advice!