‘Validated’: what recruitment AI can actually predict

Recruiters are being offered tools described as validated, scientific, predictive, and fair. These are separate claims. Evidence for one does not establish the others. But an important question to ask is: what does "validated" actually mean, and what can you do with that claim?

What is validation?

Validation is evidence that a tool can support the particular claim being made about it. The broader the claim, the stronger the evidence needed. The word is meaningless without answering the question, "Validated to do what?" At one end of the spectrum, identifying qualifications in a CV is a manageable, discrete task that can be tested and then validated. On the other end of the spectrum a much broader claim such as suitability for the role, or predicting success as an employee, is much harder to validate - to do so you must have testable data on employees already in the role.

Does validation apply to me?

If you use a tool which uses AI to affect how you progress candidates, even if it's just informing, rather than automating, validation matters to you. You don't need to be a data scientist, but you do need to understand:

• What claim is this outfit making?
• How was it tested?
• Does the claim apply to my use?

How do I get validation?

The provider should supply evidence for its claims. Your job is to decide whether that evidence applies to your roles, candidates and intended use.

Ultimately however, if you're using it, you are responsible, so you need to be assured that the tool is doing its job and is validated. To that end, ask the supplier for its evidence, and check whether that evidence applies to you. If it doesn't, that doesn't mean you can't use it, it just means you need to run a properly designed pilot with the data that you have, i.e. running the tool alongside a human process until validation is gained.

What does validation mean once I have it?

Positive validation means there is evidence for a defined claim in a defined population, within documented limits, i.e. it works for what it was designed to do. It does not automatically mean that every output is correct, so don't make that claim, but for every tool there will be an acceptable level of error while remaining validated. It does not mean that the tool is fair for every group or suitable for every role.

For example, we at Decision Agent have set the framework in place for you to progress from smaller claims to larger claims through use of the platform. Starting with the identification of behaviour within a candidate response, which is a small claim and can be validated through human oversight and accelerated with synthetic data.

We then aim to predict who you will hire - can our platform match your current HR professionals. This can be validated with parallel tracks, but crucially will only ever be as effective as your current system.

Finally we look to predict successful hires - who performs well in the role after a period of time, and can we predict that on our platform. This is the great promise of AI & automation, outperforming current pipelines on quality & fairness, not just speed & time. Without that performance data you cannot possibly validate a system that claims to predict employee performance.

Do I have to keep validating it?

You must validate again if something changes. That could be the role, the candidate population, certainly the assessment itself, models used, decision threshold, intended use, that sort of thing.

You should also just monitor whether it's behaving as expected. If all of a sudden you're getting a different result on average than you did last week, check what's changed. It could be a model update from the provider. It could be someone else's change on the variables.

Decision Agent's Position

we welcome the requirement for validation. We would rather make a narrow claim now that can be tested than a broad claim that cannot, while setting the conditions for getting the data needed to make that broader claim. Validation is worthwhile in that it will expose limitations and create a route to improvement. That is how evidence-led AI development should work. As any serious provider should be willing to explain what was tested, what was found, and what remains unknown.