Is it wrong to back out of a job that went silent for three weeks after I signed the offer?

I signed a job offer 3 weeks ago for a Technical Project Manager role at a small tech company. It had a clear start date, but after I signed, I heard nothing; no onboarding, no background check, no emails, no calls. Just silence.

I’m a federal employee and wasn’t fully committed to the role. I was applying to jobs just in case I got DOGEd. This seemed like an okay option pay wise but I learned at offer time the benefits were poor. Since they went dark, I assumed they weren’t moving forward and didn’t quit my current job.

Then late Friday afternoon, I got a random call from someone at the company saying they expected me Monday at 9 a.m. I panicked and said I’d be there… but I’m not going. I don’t feel comfortable jumping into a job with zero communication for three weeks.

Now I’m trying to figure out how to professionally decline the role this late, and whether I handled this poorly or not. Has anyone been in a situation like this? What would you do?