Am I the bad apple for not wanting to talk to a Co worker over break?

I ( 37 f) have a routine while on my breaks at work.. I normally like to just relax put my head phones in listen to music or an audio book while I drink my coffee or have a snack.. usually every one goes with that or does the same thing.

Until recently when a guy from another shift over lapped with mine.. Will call him Dylan. He (in his 40s) is just getting finished with his shift when I am on my break and he has taken to joining me in the breakdown waiting for his ride home. And he likes to chat, which would normally be fine.. but he has a very thick Scottish accent and I have a really hard time understanding him.. and a simple conversation takes a long time to get through since I have to decode what he is saying.. and then my break is over and I feel like I didn't get to really have,my break after a long awkward conversation..

I told him the next day when he tried talking to me again that I had a hard time understanding him and my break was only 15 minutes long so we could maybe talk someother time? he agreed and left me be, but I later found out I hurt his feelings.. and I felt horrible, I apologized, but then it was just really awkward I and end up avoiding him now by having my break in a different room..

So I just wonder was I the bad apple?