Companies are using AI in many parts of their workflows, but not all these experiments are working out as expected.

Hackers have allegedly managed to take over prominent Instagram accounts by fooling Meta’s AI into changing user emails. Former US President Barack Obama’s old White House account was among the ones impacted.

A video shared by X user Chetaslua shows how the hack was allegedly carried out. Users told Meta AI’s Support Assistant that their account had been hacked, and asked for help to recover the account. They then asked the AI to link to their new email address. The AI then sent a recovery code to the email address the hackers had shared, and they authenticated themselves using the code. The AI then allowed the hackers to change the password of the account, and essentially take it over.

Russia Today also shared a similar video showing the same technique.

On the Obama White House account which had 2.4 million followers, many messages were posted in Arabic before they were taken down.

Another user gave more details around how the attack allegedly happened. “The thing is the exploit is so simple it’s almost funny,” they said. “Attacker goes to Forgot Password, says their account is hacked, turns on a VPN to match the target’s location (which now you can find on the about section of the page). Instagram’s AI support flow asks them to verify with a selfie. They grab a photo from the target’s profile, run it through an AI video generator to make an animation of the person’s face moving around, upload that to Meta’s AI as proof. And Meta’s AI just accepts it because it can’t tell the difference between a real selfie and an AI-generated video of someone’s face . Once verified they change the email to theirs. Password reset link goes to their email. They own it now. 2FA gets bypassed somehow in the process but honestly I don’t know exactly how, just that it did,” they added.

OfficeChai couldn’t independently verify if this is exactly how the attack occurred, but several accounts were compromised this week. Meta has now reportedly patched the bug.

If it was indeed badly configured AI systems that caused the hack, it would raise concerns around the rapid deployment of AI systems, particularly in customer service roles. AI systems can often be overridden through threats, coaxing or coming up with life-or-death scenarios, and companies would need to make sure that their AIs with the capabilities to take account-level actions are not vulnerable to such techniques. For a company like Instagram, which has over 3 billion users, it does make sense to deploy AI for customer support, but as these alleged hacks show, these AI systems might be a bit rough around the edges at the moment for deployment at scale.