How to professionally reject a potential NF during a phone interview?

What do you do when you are on a phone interview with a family and they start talking about next steps- but you already know you don't want to work for them?

I can usually tell within the first 5 or 10 minutes of the call if it's going to be a no for me. And I'm really bad at having the guts to basically come out and say it's not a good fit to them over the phone. I'll just continue on listening to them and then asking my usual interview questions even though I know I'm not going to take the position. And sometimes they'll come straight out towards the end of the call and say that they really like me and they try to schedule an in-person meeting....then I have to do an awkward " well... let me think this over and chat with my partner and I'll get back to you" just so that I can text them the next day some eloquent message saying I'm going in a different direction.

All of this to say-does anyone have a really good verbal response to let a family down easy over the phone?