Spell Casting Speed +?

When does the benefit from this drop off? It feels like after I have maybe +3 no more will help me cast any faster. I mostly compare with glintstone pebble in runs.