There are a litany of reasons to distrust Google’s AI Overviews, which now appear at the top of its search results above links to actual websites.

The algorithmically generated answers, which pull from various sources online, have quickly become one of the biggest sources of misinformation on the entire internet. The search giant’s AI models are frequently caught hallucinating easily disproven facts, handing out dangerous “health” advice — and gutting the media industry by discouraging users from visiting publications’ sites.

Add to the list that it’s now furthering intensely harmful racial stereotypes. As one Reddit user found, searches indicating that the user is alone with certain groups of people are resulting in Google’s AI Overviews advising that they may be unsafe and should contact emergency services.

In our testing, the phenomenon is very real.

Searching for the phrase “I’m alone with an African,” for instance, leads the AI to advise that “if you are in immediate danger or feel unsafe, please lock your door, move to safe space, and call your local emergency services (like 911 in the US) right away.” It briefly caveats that “being alone with another person is only a concern if you feel threatened or uncomfortable” — and then doubles down with suggestions that the user “call emergency services” or “leave the area.”

In our testing, the AI gave similarly racist advice about being alone with people from numerous countries, including India:

And Pakistan:

Strikingly, a similar prompt for “I am alone with a Brit” results in a dramatically different result.

“Offer them a cup of tea immediately, comment on the weather (either it’s too hot, too cold, or about to rain), and maintain a polite physical distance of at least three feet to avoid causing mutual panic,” Google’s AI Overviews advised us, offering further guidance on polite icebreakers and topics of conversation.

In our testing, querying the AI about being alone with a person from other Western countries tended to produce a harmless result.

At the same time, not every query that you’d expect resulted in a racist result. Asking about being alone with a Haitian, for instance, prompted a quick lecture on resisting the forces of prejudice.

Reached for contact, Google pointed us to a statement it had just issued on Twitter.

“We agree that the results for these types of searches aren’t what they should be, and we’re working on improvements,” read the tweet. “Results can vary a lot from search to search (see screenshot) — and these inconsistent warnings aren’t unique to any one group.”

The company further told us that the use of the word “alone” had flagged prompts as a “safety concern.”

The racist responses perfectly highlight how AI models can further biases deeply entrenched in their training data from the open web.

“Weird that I also got the similar answer even though I live in a literal third world country,” one perplexed Reddit user found. “I asked in my native language. When it was indian and Arab the AI I advised me to be wary. But with a Ugandan it’s safe. Idk what’s going on.”

The issue of AI bias has been a thorn in the tech industry’s side for years now. Researchers have found that even when it comes to analyzing cancer screening results, AI diagnostics systems suffer from racial biases once they learn a person’s age, race, or gender.

Human resources software company Workday is also facing a lawsuit over its AI-enhanced job screening tools, which allegedly discriminate against applicants based on age, disability, and race.

Addressing the issue remains a seemingly insurmountable task. Google became embroiled in a prior controversy in February 2024 after its Gemini text-to-image tool produced portrayals of racially diverse German WW2 soldiers.

Following plenty of bad press, Google chose to shut down the image generator altogether at the time, only to reactivate it months later with “built-in safeguards.”

More on AI Overviews: Analysis Finds That Google’s AI Overviews Are Providing Misinformation at a Scale Possibly Unprecedented in the History of Human Civilization