Regretting leaving BIG 4?

I'm thinking about leaving my current job at a big 4 firm and I wanted to see if people have regretted making this decision? I would be leaving it for a smaller public firm.