Anyone else here prefer to spend their work lunches alone?

I just don't feel the need to socialize, and I prefer my solitude by myself versus constantly going out with my Coworkers to eat. They nearly push for it twice a week (well one guy initiates it) since we all work in separate buildings with the same role, we hardly see each other unless in passing. This is not to say that I always exclude myself, I usually make myself every now and again, but when I'm mentally overwhelmed from work and they (again he) directly pushes and asks if I'm going to go, I either make an excuse or something up later feeling bad about it. Anyone experienced this dilemma?