Sanding - How to Get Rid of Scrapes / Left Over Residue

I recently sanded down the pegs on my SS86 Devastator’s chest piece to negate worsening the stress marks that had unfortunately formed on my copy. They fit much better now, for some reason they lost their click when tabbing in, but they go in much easier, but they have this weird residue that won’t go away when I wipe it down, I clean it off with a wet wipe then it fades back in like nothing happened, it’s a little freaky. To make it clear I didn’t use slightly increase the level of grit over time as I only had one sandpaper box on me at the time. Is there any way to properly clean it off so it’s a little less unsightly? I know it will be barely seen but i want to know if this is fixable, especially if I sand again in the future. Thanks for any help anyone can provide! My best guess is the plastic has a new grit and is picking up residue from tissues and wet wipes.