Should I be worried about my coworkers behavior towards me?

I just started my first job this April. It’s a manufacturing company and I am working in research and development. I don’t have JLPT qualifications but can speak, read and write enough Japanese to get by. But nevertheless, it gets really tiring to pay attention in the meetings and trying to simultaneously translate everything in my head.

My coworker is a very sincere guy, i had noticed him during the training too. He always asked a lot of questions, because of which some guys used to make fun of him. Since i only hanged out with other female coworkers only ( i am also a woman) i never got to know him well until we joined the same laboratory.
He is very sincere in the sense, always taking notes and asking questions. While my priority is usually to understand as much as i can and then read the documents by translating them to english later, to save time. (Before anyone attacks me, i am improving my Japanese vocabulary everyday).

Now we get 1 hour lunch break in a 9 hour 45 minutes job. Except for me and that coworker, everyone else is seniors who have wives at home to cook breakfast for them. (Yes their wives are working women, yet take the full responsibility of house chores, they brag about it🤢) so naturally they wait until the cafeteria is less crowded, so they can go and have lunch.
I usually don’t get time to eat any breakfast. I wake up at 6 and the lunch time is 12:00 so I prefer going to cafeteria as soon as it’s 12.
I am also the type to prefer eating alone because I don’t wanna talk in Japanese at least during my break.
My coworker waits until 12:30 so he can go with all others in the lab, and he has been taunting me about not waiting for everyone.
Why should I waste 30 minutes out of my 1 hour break in the lab where i will be sitting for another 4 hours after break??

When it’s time to go home, i leave immediately but he waits for other members to finish.
I try to reach home before 6 PM because i cook everyday and i want to have a life outside office but my coworker keeps saying that it’s insincere and rude to leave at the exact time.

What should I do?