When are you buying nursing bras?

I’ve noticed a few maternity/nursing bra sales pop up, but is it possible to buy bras too early in pregnancy? I’m currently 24 weeks pregnant, and I noticed a pretty drastic change in size early on in pregnancy, but not much has changed up there since then. Will they grow again in the third trimester? Is it best to wait until as close to the due date as possible to get the right size?