[Routine Help] Do I need to wash my face after shampooing my hair?

Hi, I tend to wash my face twice a day. Once in the morning after my shower and once at night before bed. I believe my skin may benefit from only washing it at night but my dilemma is this: Any shampoo or conditioner residue that may be on my face from my morning shower. While I try not to let it touch, I know it does, especially my conditioner because I let it sit while I shave my legs.

So my question is: Should I keep doing as Im doing or do you guys think I would be fine to only wash at night, even with possible residue?