Upgrading my phone.

I currently have a note 3 (Broken). I am looking at changing to Verizon (From t-mobile) and I am considering upgrading phones. The note has been my favorite phone by far (Besides the never setting preferences). I have looked at the note 4 and 5. People seem to dislike the 5 though the main reasons I do not really care about. I don't know if I should. I have never needed to remove my battery, or use my sd card. Is there a big reason to side with one or the other? Or is there another phone that you might suggest?