The Gabriel vs. Collins era Genesis argument

It's generally quite well known among the Genesis (rather, the whole Prog community as a whole) community that there are a lot of fans divided on things that are post Peter Gabriel, or vice versa, with some regarding anything behind A Trick Of The Tail as bad, unoriginal, not their thing, what have you. On the other hand, we have a large amount of Genesis fans, many being new to the band, who believe the leaving of Peter Gabriel sounded the loss of quality in the band and its "degradation towards pop" (myself having been one of these people previously, before I on a whim listened to Invisible Touch and was blown away). Is there any reason that this should be a huge thing? Has it been going on for decades, ever since the split happened? Is it even worth it limiting oneself from all the amazingly creative works of Genesis post Gabriel?