[No Spoilers] Are season 4+ good?

I finished season 3 and a lot of people tell me it's not good in the later seasons. I'm not a serious fan or book reader and tbh I can't remember most of the characters names, but GOT was probably my second favorite show ever.