Colorless pencil blender is changing the hue of single colors

I use Polychromos pencils and they don't have a colorless blender, so I got one from Caran D'ache, it works pretty well to smoothen out an area, but it's changing the hue. I'm not talking about color mixing; using a single color then using the blender to burnish the paper texture changes that color. Tried with different colors and they all change hues. Any idea what's happening?