Should I take my cat with me?

I live and work in Washington, but I'm from Mexico. A couple of years ago, my brother (in Mexico) found an abandoned kitten, so he took him home and adopted him. He's a happy cat, but I often feel like my brother doesn't really care for him. He's often neglected, even though my brother constantly claims "he's doing fine". He's an outdoors cat, but has access via a kitty door to the inside of the house. I've considered bringing him with me, since the cat seems to like me (he hangs around inside the house a lot more often whenever I visit; after I leave, he hangs around my room and meows for days). The only issue with taking him with me is that I live in an apartment, which although it's decently sized, he would no longer have access to outside, and he would no longer be able to roam free. What should I do? Take the cat with me, or leave him where he is?

Obligatory cat pics. Both taken on either the first or second day I came back home.