[NY] Doctor at work with reputation of sexual misconduct increasingly flirtatious and gives me preferential treatment. How to navigate?

I (30 f) work in a hospital and I’m dealing with a complicated dynamic with a psychiatrist (54m) that I work side by side with. For some background:

I am new to the job and career for that matter and I’ve now had 2 people pull me aside to tell me to be “careful” of him as they have seen his behaviors with me and one person has told me he has a history of misconduct. He used to be the medical director a few years ago, but lost his job and got divorced because he had an affair with a resident (student) he was supervising. She reported him and told him wife and he left the hospital and lost his title as medical director. From his behaviors with me, I can only assume he is very experienced with this type of behavior and I doubt he only had one affair with one student. Now he’s not allowed to work with residents because of this - since it’s a liability to the university they study at. (He also worked at the university and lost his job there too because the student reported him)

Now, he’s back at the hospital for some time now. (Before I got there) He works part time and he is “in charge”on the days he works there, and he chooses me to work with. He gave me a lot of compliments when I first started on how well I’m doing and lots of flattery which gave me a big boost of confidence. But slowly, I started to notice he would tease me about my clothes , sneakers, make it a point to let me know he pays attention to me “what was with your face in the meeting today? Why are you so tall today!?” “Why weren’t you in the office last week!?” He is actively flirting with me, gives me preferential treatment (I won’t give you this case because I don’t want you to work too hard….”I should give you this one case but you know what? I’ll wait until [coworker name] comes in and I’ll give it to her.” “You know who’s in charge tomorrow, I can make it happen that you don’t have to do xyz tomorrow”) and is strategic about how he interacts with me — acting playful but professional in front of others, he will change the subject if my direct supervisor walks by and will say “good job today. Good work” etc but being flirtatious or overly familiar in private in his office, behind closed doors, but will not initiate anything.

He’s good at making me feel special and in the beginning I really did feel special because he gave me many compliments and seeks extra time to train me and teach me about medications etc. he also shares his movie, sports etc interests with me. I thought we had a good friendship and rapport and connection but I realize that’s not what it is now. I myself can have a playful personality and engage in banter and I used to do that with him. He’s good at lowering my defenses and my guard because he himself is playful and quirky.

This pattern makes me uncomfortable, not only because of the attention itself but also because I’m worried others may notice the favoritism and assume something is going on. He has a past history of inappropriate workplace relationships that cost him a leadership position, and his marriage which makes me even more cautious.

The hospital protects doctors and I don’t know what to do and how to go about this situation since he hasn’t directly forced or done anything to me. But there is an obvious power dynamic here. I’m afraid to tell the new medical director about this because he may want to move me to a different department or change my shift so I don’t have to interact with him but I like my job and I like my work hours and I don’t think I should have to be the one to change things around, especially since he is liability. Is there anything I can do about this situation?