Why do we feel guilty for doing nothing, even on days we're supposed to rest?

Like, I took a full day off last weekend, no work, no chores, just watching TV and laying around. And the whole time I felt this nagging guilt, like I was wasting my life. But isn't that the whole point of a day off? Why