Just a quick question!!

I always had this question in mind, is it necessary for context size to be higher than chat memory in order for the ai to remember everything?? I like long RPs and some of them have 50k chat memory so I was wondering if I put my context size up to 60k would there be any downsides?!