Well, shoot

Edit: Spoilers for Dr. Xu romance

I am happily married to Arlo, but tonight I had the firework meet up with Dr. Xu. I didnt realize it was romantic, and when he asked to become more than friends I panicked and accepted.

I did a LOT in game this day and would hate to lose the progress. Before I restart the day I need advice: what would happen if I kept that sweetheart status with the Doc but didn't pursue it further? Does the spouse get angry or does no one notice as long as you don't interact with said sweethear?