Why do people keep saying that's not Lilith?

I keep seeing people theorizing that the lady ignoring Charlie's phone calls isn't actually her mom, and I'm just wondering why people are thinking it's not her bc as far as I can tell there's no real reason to believe that. Like we know hardly anything about Lilith as a character, she hasn't even had any voice lines yet, idk it seems less of a leap to just assume she's that shitty and that Charlie and Lucifer are just seeing her through rose-colored glasses and/or she's been hiding a LOT from them