Job Questions and Tips

Hello (M 27)
I was let go from my long term job of 5+ years in mid may.
I had some decent interviews but nothing stuck. I was getting desperate - I finally landed a job paying slightly less than I was making. However this new position really sucks in terms of benefits, little to no paid holidays (1 day a year) 8 pto days and 8 sick days. Along with my schedule while 8-5 can rotate days and is not set M-F. I would be starting Monday July 27th.
My new issue is - I’m finally starting to hear back and get 2nd or even 3rd round interviews from companies I’d much rather work for.
Should I reach out to this company I’m about to start for and see if I can push back the start date? Or even just let them know I won’t be moving forward with them? Because truthfully the minute I get an offer from another company I’d quit.
My hesitation comes from a place of possibly missing out on interviews because I’m working the new job. Along with while I know I owe this new company nothing I do feel slightly bad about starting and then quitting shortly after.