0.1% retinol from TO causing irritation

I decided to incorporate retinol into an otherwise very simple skincare routine to increase turnover (I've been getting a lot of milia, blackheads and fine lines). I started with The Ordinary's 0.1% retinol in squalene. I used it every day for a week (I know, I know) and my skin got really textured and a little inflamed. I stopped using it for a couple of weeks and my skin recovered quite a bit. Now I've restarted it and use it only twice a week, but my skin has returned to being textured and inflamed. Should I take another break and try again, try a different retinol/retinoid product or try something else altogether?