As with the CV scoring, the way in which spoken responses are evaluated was not explained. It is unclear exactly how a qualitative result was quantified.

The AI interview bot also glitched in several interview sessions for various test candidates. On some occasions, it did not ask the next question until the candidate had to prompt it. Other times, it repeated the same question twice in a row.

This technical bug could influence a potential candidate, and might be more awkward answering a second time, or suggest a worse performance compared to a candidate who did not experience the glitch. Moreover, it is unclear, if a question was repeated, whether the first or second answer, or a combination of both, would be factored into the AI system’s final score.

A further concern is that, at the time of our testing, there was no facility for a human recruiter to override the scores generated by the AI interview - nor was the human recruiter required to intervene and approve or reject the AI interview score before it was assigned to the candidate’s result. The result of the interview as assessed by the algorithm, and the AI match score from the CV screening stage, are the only assessment results on the platform. While the platform does provide video recordings of the interviews and AI-generated transcripts of the interviews, and the human recruiter can manually move the candidate to the Reject or Accept folder, there was no ability for the recruiter to edit the candidate’s match score (e.g., if the recruiter reviewed the transcript and video and wanted to rescore it themselves).

Talenteria has clarified in their response to us that the platform has since been updated, and the current version now has the ability for a recruiter to override the AI Match Score with their own scores. Manatal did not provide a response.

We welcome Talenteria’s addition of a manual override function. The lack of human override functions is a risky design nudge that could potentially harm a candidate later down the line if they are passed onto a different recruiter to manage, who may be confused about why a candidate who has such a low AI Interview Score was filed into the Accept folder - was it a mistake? Might the new recruiter be more harsh in their review of this candidate due to anchoring bias?

Even with an override function, the algorithmic score is still the first, default score for a candidate, though. The more important question here is whether platforms are designed to encourage, if not require, human recruiters to correct the AI’s score or add their own score upon human review, or whether platforms are instead designed to favour the default algorithmic recommendations.

What’s going on? Criticisms of transparency

While both platforms offer an AI-generated analysis of why the platform came to its judgment of each candidate, there was no easily available explanation of how qualitative information from the CV is translated or calculated into a quantitative output and no AI weighting publicly disclosed.

Both Manatal and Talentaria produced written AI summaries on how the candidate matched against the job application (these summaries varied in language across identical candidates). For example:

• “Candidate meets two required criteria, including graduate degree and human rights knowledge, but lacks experience in key areas like sysadmin and project management, with no preferred criteria met,” read one Manatal summary for the Technologist position. The candidate received a 23% match score.
• “The candidate has less than 3 years of relevant experience in legal roles, which is below the 7-year threshold for a higher score. Their experience is somewhat aligned with the job responsibilities but lacks depth in human rights and legal advocacy,” read one Talenteria summary for the Legal Officer position. The candidate received a 7/10 match score.

However, there was no explanation for the breakdowns, nor did the platforms provide explanations for how the quantitative scores were calculated from the qualitative information on the CVs.

This might make it difficult for a recruiter to explain the logic behind their decision to the candidate. It might also make it difficult for a candidate to exercise their data subject right to correct information about themselves or, potentially, their GDPR Article 22 right not to be subjected to a decision based solely on automated processing, if a recruiter were to use the AI’s decision without making any changes (i.e. rubberstamping) or without providing their own meaningful human input.

Talenteria clarified in their response to us that:

“Talenteria is not intended to be used as the sole basis for hiring or rejection decisions. Employers remain responsible for their recruitment process and decisions” (Annex A).

Crucially, according to the UK ICO, “the degree and quality of human review and intervention before a final decision is made about an individual are key factors in determining whether an AI system is being used for automated decision-making or merely as decision-support.”

The inconsistency of scores for the same candidates significantly impacts fairness in the recruitment process. This inconsistency could partly be due to faulty or otherwise incomplete technology at the time of our testing, and Talenteria informed us that they update their workflow development frequently:

“On average, we release a new product version approximately once per month, with improvements that may include AI model updates, scoring logic improvements, user interface changes, interview workflow updates, and customer-requested enhancements” (Annex A).

However, there is a fundamental problem with constantly updating models at such a rapid rate. While the intention may be to use the most updated model release in the interests of improving the platform’s capability, candidates who were subjected to the older model’s assessment are at a disadvantage. If new AI models are being released and adopted that may score candidates differently from previous iterations, this could be unfair to candidates who happened to submit their CV before a platform update. It’s not clear whether these updates might occur mid-application cycle or at the close of an application process. Even if platforms were to disclose to the candidates that a model has since been updated, this retroactive transparency disclosure does not remedy the impact to fairness.

We were also unable to test from the candidate’s perspective what internal mechanisms these platforms offered for candidates to understand, correct or challenge their result, should they feel they have been unfairly rejected. It is possible that this limitation could be due to the free trial; nonetheless, we could not find further information on Manatal’s or Talenteria’s websites about how candidates could challenge their AI score.

Conclusion

Our investigation found that inconsistencies in assessment or interpretation, technical glitches, or simply the inherent unreliability of the algorithms through issues such as hallucinations could produce an unfair outcome for a candidate.

The inconsistency in scoring should give pause to anyone relying on these platforms to make fair hiring decisions. Furthermore, the preference these systems seem to show for AI-generated CVs compounds the concern, especially in cases when employers might discourage the use of AI in preparing application materials.

If these platforms are indeed rewarding keyword optimisation over genuine qualification, it is hard to see how this technology serves the interests of recruiters seeking the best candidates, or of candidates who present their experience honestly.

Of course, human reviewers come with their own levels of variability and bias, but the issue we are raising here is not on recruitment in general, but on the reliance and use of algorithms to influence recruitment decisions. Algorithms must be shown and known to reduce, rather than embed, risks of error or bias into important decisions. Oftentimes discussions around AI recruitment tools veer on the apocalyptic ‘AI replacement’; however, we scrutinise even the level below it of AI-assisted decision-making. AI-assisted tools can nudge and influence human recruiters to make certain decisions based on an algorithmic assessment.

Furthermore, even if human recruiters may potentially judge things differently from one another, at least they can explain their reasoning behind their decisions, and they also avoid the risks that AI introduces that we’ve shown above (e.g. nuanced assessment of short answer questions).

When a candidate’s application is evaluated by an algorithm, the criteria, weighting, and logic of which are hidden, the candidate’s ability to understand what is being done with their personal data - and why it has produced a particular outcome - is undermined.

In addition, it is possible that as job applicants use AI to write their cover letters, with recruiting platforms such as these in our experiment inadvertently rewarding such action, weaker candidates’ hiring rates can increase and stronger candidates’ success rates fall.

Data protection law (at least in the EU and the UK) gives candidates various rights, such as to make subject access requests, to correct inaccurate information, and to be provided with meaningful information about the logic of automated decisions. Article 22 of the GDPR even places limits on the use of automated decision making for significant decisions in the first place. Taking human decision-makers out of crucial parts of the recruitment picture makes complying with these rules challenging. Without visibility into how an application is processed or documentation of the system’s logic, a recruiter may struggle to meaningfully explain their decisions and a candidate will struggle to exercise their rights.

Candidates can make Data Subject Access Requests to platforms to seek more detail about their recruitment process, but the inaccuracy and incoherence of the AI-generated scoring and feedback could put employers at risk if they disclose that their hiring decisions were based on inconsistent scoring systems. It is also difficult to say whether the information recruiters can provide is comprehensive enough to meet the legal requirements for explainability, due to the opaque nature of the technology.

The AI scores recruiters see carry an air of authority — a precise percentage, a star rating, a written breakdown — that implies objectivity and reliability. But the reality is that those numbers may shift arbitrarily between identical submissions, and that the written explanations provided by the algorithm do not provide insight into how or why the algorithm scored a candidate in a particular way.

If AI is going to be used in recruitment, it should provide more meaningful transparency about scoring functions, and about how candidate decisions are made. Otherwise, employers risk systematising recruitment bias, making it less visible beneath the veneer of the black box machine. Meaningful human oversight and intervention is also necessary to ensure that issues with an AI recruitment platform do not remove good candidates from contention.

The findings of this investigation present a troubling picture with growing industry reliance on outsourcing consequential human decisions to systems that risk being unreliable and opaque. While our research concerned these two platforms, they merely serve as case studies; the problems we’ve identified are rife and may well be replicated among other automated screening platforms, such as we’re seeing with further frustrations on AI interview platforms and scoring platforms.