Dentist charging for refinement

I paid for a 'full' Invisalign package (20+ aligners) and in this quote it says refinement will be charged if aligner wear is not optimal. I have been very compliant wearing my trays. A few months in, I noticed 2 teeth not tracking, I was told to keep going. I kept bringing this up in every subsequent appointment and I was told to keep going by the dentist despite the gap increasing and attachment not fitting the aligner. Now I'm near the end, he finally said refinement is needed to address these teeth and wants to charge me for refinements. Should I be paying? I feel like I have done exactly as advised and wore my trays as required