Does anyone feel guilty having to say no to side babysitting?

I recently babysat for the first time for a new family who I really liked / was hoping to sit for again.

They know that I nanny for a family and sometimes because of that my schedule conflicts.

They had asked me if I’m available later this week and unfortunately on the day they asked me it does conflict, also sometimes I am tired from the workday so because of that I declined.

I was feeling really guilty afterwards and I don’t know if this is typical, I guess because I only babysit them one time so I don’t want them to think I don’t want to going forward when I am available. Does this ever happen to anyone else where you get side babysitting but feel kind of guilty for not being able to take it?