AI + ML

Stanford researchers argue need for transparency and independent testing

AI algorithms exhibit racial bias in job candidate screening, and they discriminate more frequently against those applying for multiple jobs at different companies, according to Stanford-led researchers.

The boffins evaluated algorithmic hiring decisions across multiple employers that use the same hiring vendor. The resulting algorithmic monoculture, they say, is problematic.

The vendor in this instance was talent platform pymetrics, acquired by Harver in 2022. Harver did not immediately respond to a request for comment.

The researchers – Rishi Bommasani, Sarah H. Bana, Kathleen A. Creel, Dan Jurafsky, and Percy Liang – obtained a pymetrics dataset spanning the period from December 2018 through December 2022. It contained 4,197,168 job applications submitted by 3,372,132 applicants to 1,746 positions.

The dataset details hiring recommendations provided to 156 employers with a total annual revenue of $225 billion. It spans 11 industries, including finance, manufacturing, and warehousing.

When people applied for jobs at these companies, they were directed to pymetrics' machine learning platform to play assessment games. The platform's algorithm measures gameplay performance and recommends on average 58.2 percent of applicants per position. Employers decide who to interview, typically rejecting candidates who were not recommended by the hiring platform.

The researchers contend that the pymetrics algorithm is unfair.

"We find substantial evidence of racial disparities in AI-based candidate screening," the researchers said.

They made that determination by applying the US Equal Employment Opportunity Commission’s "four-fifths rule," which at least on paper elicits agency attention when a given group's hiring selection rate is less than 80 percent of the most recommended group of job applicants.

"We discovered that 26 percent of Black applicants and 15 percent of Asian applicants applied to positions where the AI system discriminated against their racial group," the researchers said.

If those Black and Asian candidates had their job applications advanced at the same rate as the most favored group (commonly White applicants), about 40,000 more job candidates would move on to the next screening stage.

What's more, the report authors say that when people submit multiple applications at different companies that use the same hiring algorithm, they're more likely to be rejected everywhere than if the companies used different hiring technologies. They found 10 percent of job seekers who submit four applications were rejected from all the places where they applied for jobs. 

That pattern they say does not show up in hiring studies that look at hiring without regard to the use of AI – there the rejection rate fits what would be expected if every company made its own decisions rather than relying on a single algorithm.

The study authors note in their paper [PDF] "Algorithmic Monocultures in Hiring" that prior work has documented discriminatory patterns when decisions are based on applicant resumes (e.g. when names or activities are more common among certain groups). 

The game-playing approach used by pymetrics may lack that sort of demographic information, but the researchers say they find adverse impact despite the absence of demographic details and pymetrics' efforts to de-bias applications.

They say their findings support prior work showing that AI can have discriminatory effects even in the absence of demographic data because the AI models zero in on variables that are proxies for demographic data (e.g. when a demographic group is overrepresented in a particular zip code or at a particular school).

When pymetrics researchers examined the impact of AI for hiring in a 2022 paper, they found that their algorithm would not run afoul of EEOC standards. They argue that fair hiring is complicated and that candidate selection prior to the advent of AI also had problems.

"[W]hile it is true that machine learning can introduce harms in the form of systematizing bias and obscuring discrimination, these effects are already pervasive due to widespread use of traditional assessments in many industries," the authors of the pymetrics study said.

The Stanford group attributes those results to pymetrics' approach, which involved pooling all its recommendations and considering them in aggregate. The discrimination doesn't show up when it gets averaged out. It's necessary, the authors argue, to consider jobs separately.

"For example, imagine the AI tool frequently recommends Black applicants for warehouse jobs but rarely recommends them for finance jobs," they explain. "If we were to average all the jobs together, those two patterns would cancel each other out and it would seem like there is no discrimination. The big-picture average hides the real discrimination happening job by job." ®