I only just figured out that clicking a downvote that I made removes it.