Is it more rude to accept or to refuse gifts?

I feel DEEPLY uncomfortable accepting gifts, like they should just give it to someone else or keep it for themselves, but i’ve been told that’s super rude so now i’m confused.