"Mama I love when you go to work"

My almost 3 year old son said that to me right before I went to work yesterday. I've been fighting guilt and sadness over spending so much time at work and I know my son is starting to notice when I have to work late nights. So when he said that to me, it hit me right in my darker thoughts that he likes it better when I'm not around and I've been thinking about it ever since. In comparison, earlier this week, he told my husband he loves when he puts him to bed, and just now he told my mom he loves when she comes over. Not at all sure how I am supposed to take "Mama I love when you go to work" in comparison.