Avoiding contact with partner with a flu.. yes or no?

My partner had flu and asked if we should not sleep together and keep distance so he doesn't infect me, but I downplayed the risk.

Now I'm sick and feel bad that I should have listened to him, but my logic back then that I didn't want him to feel rejected or I'm scared of him. I don't know what is the right thing to do, how do you deal with it?