forgot increases - flat panels

Hello! I am doing a pattern with an intarsia colourwork image in the centre and missed the line in the pattern where it says “start the armhole shaping alongside the colour chart”. So have done the colourwork and am reluctant to frog it back as it was frustrating me so much to do at the time 🥲

Is there a way to fix this without ripping it all back? Could I knit two little wedges and mattress stitch them in on either side instead of the increases? Or do I just accept my sad fate….