Sidequest Priority Question

Hello! Playing through the Definitive Edition, it's my first time through the game and I just have a question about all the non-story quests in the game...

The way my brain works has me doing every available affinity and green quests while regularly rotating party members to raise their hearts while doing their heart to hearts as they become available, all before moving to the next story mission (I understand this is a weird me problem, not looking for criticism or supportive/dismissive "have fun your way!" comments).

I'm now level 32 and just finished chapter 6 and got my skells and I'm burning myself out with my sicko meticulousness - if I start to just focus on exploration/story/whatever while using a set party, will I always be able to go back and do all of the other side content (barring the character you eventually lose) and easily raise affinities without significant penalty/missing things? Thanks!