We're about 6 demos deep into evaluating AI application review tools and honestly they're all starting to sound the same

Every vendor says they "use AI to screen resumes" but when you push on specifics it gets vague real fast. So before I burn another week on calls I built a quick eval rubric to actually compare these things properly, this is what I'm scoring against so far: ICP-driven scoring where the AI scores candidates against the actual criteria we set with the hiring manager, not some generic resume quality grade. Rationale per candidate so I can see why someone scored a 4 vs a 7, not just a number. HM override where hiring managers can disagree and the system learns from that feedback, and ATS integration so it lives inside our existing workflow with no separate tabs or copy-pasting, also have an audit trail showing who saw what candidate and what decisions were made, defensible if we ever get challenged.

I'm sure I'm missing things though, bias testing and adverse impact reporting feels like it should be on here, same with how the AI handles non-traditional candidates like career switchers or people without degrees. For those of you already using something in this space, which of these criteria do the tools actually deliver on vs just claim on the website? The demo is always clean but I want to know what breaks at scale, what surprised you once you were in production with it.