Am I doing seedlings wrong? (US Zone 5b)

I started off some tomato seeds in one of those peat seedlings starter greenhouse trays you get at any garden center. That was on March 14th. Out of the original 8 pods I started, only 3 of them have survived.

All 5 that died off withered out. At first I thought it was due to the seedlings stretching out due to our light being too far away, but even setting up a grow light 4 inches away still had several of them die off. Did they just end up getting too big for their peat pods and actually needed to be transplanted to a bigger pot? Or could it have been an issue of them not being fully trellised?