Eyeshadow/liners causing constant irritation?

I've been wearing makeup regularly since I was a teenager, but in the last year I've really simplified my look: concealer, brows, lashes are my everyday. Now, whenever I add a eyeshadow primer, eyeshadow, or eyeliner I get irritated red, swollen skin along my lash line within a day or two. Mascara alone very rarely causes this. I think I've made my body more sensitive and less tolerant of eye makeup!

I've talked to my doctor about it, but her suggestion of exfoliating with baby shampoo only aggravated it worse. Once I switched back to my regular sulfate free facewash and got really strict about how and when I washed my face, it got better (only on a daily, minimal makeup basis, more eye makeup still causes issues.)

I don't think it has anything to do with the quality or cleanliness of my shadows and liners. I get it with my "tried and true" as much as with brand new stuff. I don't ever sleep in my makeup, in fact I usually have it on for less than 12 hours. I miss my makeup :(

So my question is, if anyone has had a similar issue, what did you do?