I'm an altar server and i sometimes feel bad about handing out communion.

I am an altar server at my church and because i'm one, the priest sometimes lets me hand out communion beside him. I feel like i'm doing something wrong and that only the priest should be allowed to hand it out. Should i keep handing out communion when the priest asks me too or should i not do it? I wanna ask my priest about it too but i also want to hear about what you think.