People get mad when you take leave?

Why do people always say “take your leave!!!” Then get upset when you use it?

My supervisor was literally giving me weird vibes for taking leave again ? And I was like “you’re just upset you don’t have leave to use” and he kinda jokingly agreed.

But what is up with that? Why do I feel discouraged to use leave? Why are people like that??