If hard work is the key to success, why do some of the hardest-working people stay poor?

So I've always heard people around me say that if you work hard enough, work long enough hours, you'll be successful. But when I look around, especially in the area that I live in, I see a lot of people working incredibly hard... I see lots of construction workers, caregivers, warehouse workers, people working multiple jobs and more, and many of them still struggle financially. But I also notice some people seem to make much more money while working fewer hours or sometimes not at all. How?

I'm not asking this to start an argument or anything I'm genuinely curious how people explain the relationship between hard work, opportunity, education, luck, connections, and success. I would also love to know what you guys think without having to bring race or ethnicity into it.

Is hard work really the main factor, or is it only one piece of a much bigger puzzle???