How do you know when it’s time to leave your firm?

I’m actually at a US firm in London, but the uk law sub is pretty much overrun with students so posting here. I’m a mid level associate and I trained and qualified with this firm, so I’ve been here for almost 5 years total. The hours naturally aren’t great but I really like the culture and the people as a whole, and generally get on really well with my colleagues, who are all pretty nice and normal people as far as firms go.

The problem is the work - without getting super specific, my wider team has decided to specialise in an area of work that I’m not interested in and have no prior experience in. Basically, I qualified into X team but X team has no work, so we have to do the work of Y team but we’re not ‘officially’ in that team. We’ve had a few people at my level leave and I’m starting to feel like a body to be thrown at random matters. My work preferences are literally never taken into account (and it’s not just me experiencing this).

My appraisals are always positive with absolutely no constructive feedback, but I don’t have an influential partner sponsor and it feels like there are a select few chosen associates who are focussed on. In-house isn’t an option for me yet. I know I could coast on my goodwill here for another year or two before leaving, but the problem is that I would be too specialised to pivot outside of this niche practice area we’re all focussing on.

I’ve started speaking to recruiters but haven’t made the jump yet. I know other firms wont offer more money or better hours, but they might offer the work I actually want to do. When did you know it was time for you to leave your first firm, and do you regret leaving when you did?