Remote app won't connect to  TV

I've tried connecting the remote app to my Apple TV, however when I tap new device it won't show up on the Apple TV, or if it shows up, it will disappear while I'm typing in the code. It's really frustrating. Does anyone have a solution?