Get rid of one elder rift modifier

Can you just get rid of that elder rift modifier that drains life each time you use a skill? It is simply annoying. It doesn't make the elder rift more challenging or easier, it just makes it more annoying. Waste of time.