I've finished reading the Draft Guidelines for transparency requirement under the AI Act that are out for comment until the 3rd of June, and a variety of exemptions for law enforcement and similar actors greatly concern me. I haven't seen media pick this up in any meaningful way, but this should be highlighted and discussed.
A short explanation upfront:
Transparency requirements under Art. 50 AI Act refer to providers (and some deployers)1 of AI systems intended to interact directly with natural persons needing to make sure that the users are informed about interacting with an AI system, and outputs being marked in a machine-readable format and detectable as artificially generated or manipulated. That covers, for example, AI-enabled voice assistants, chatbots in various settings (even on social media), (humanoid) robots and AI companions, AI avatars, coding agents and agentic AI systems.
Depending on the provision, transparency can be done via direct disclosure to users (such as banners, pop-ups, notices, voice announcements or chatbot messages), or machine-readable marking and detectability mechanisms for AI-generated content, sometimes complemented by visible labels or watermarks. Simply stating it in Terms of Service, documentation or else, or having a non-visible watermark, is not enough to inform users. This needs to happen at the very first interaction as latest point. Obviousness-exceptions apply.
Throughout the document, law enforcement and related actors get several exemptions, starting with 3.2.2 Exception for AI systems authorised by law for law enforcement purposes, points 43-46, page 15, emphasis mine:
Providers of interactive AI systems are exempted from the transparency obligation under Article 50(1) AI Act, if they are authorised by law to detect, prevent, investigate or prosecute criminal offences, subject to appropriate safeguards for the rights and freedoms of third parties.
[...]
To fall within this exception, the purpose of the AI system must be to detect, prevent, investigate or prosecute criminal offences (e.g. AI-undercover agent). The exception is not restricted to the use of such AI systems only by law enforcement authorities as defined in Article 3(48) AI Act, but may also cover interactive (or generative) AI systems used by other EU or national public authorities or even private actors, such as security companies or financial institutions, so long as their use is authorised by law to detect, prevent, investigate or prosecute criminal offences and subject to appropriate safeguards to protect the rights and freedoms of third parties.
Or point 87, 4.3. Exceptions to the obligations under Article 50(2) AI Act, page 23, about labeling and detection:
Finally, if a generative AI system is authorised by law to generate or manipulate synthetic content to detect, prevent, investigate or prosecute criminal offences, it will be exempted from the marking and detection requirements under Article 50(2) AI Act.
Or point 103, 5.2. Out of scope, page 26, for emotion recognition and biometric systems:
The obligation does not apply to emotion recognition systems and biometric categorisation systems that are permitted by law to detect, prevent or investigate criminal offences subject to appropriate safeguards for the rights and freedoms of third parties and in accordance with Union law.
Or point 117, 6.1.4. Exception for law enforcement, page 31:
If the use of a deep fake is authorised by law to detect, prevent, investigate or prosecute criminal offences, deployers are fully exempted from the transparency obligation under Article 50(4) AI Act.
The way it looks right now, AI systems used by law enforcement (and related actors, like security companies!) to detect, prevent or investigate crime will be exempt from several core Article 50 transparency obligations, meaning any labeling, marking or disclosure that you are interacting with AI or that you are seeing deepfake content when it is used against you.
As it stands, this enables the use AI chatbots posing as real people against investigation targets without having to tell them, and permits the use of synthetic or deepfake-style content towards targets without having to label it as such. The only exception: The bot is available to the general public and offers functionalities for people to report crimes (meaning: a police chatbot recording your complaint, virtual assistants for witness statement collection, or an AI fraud reporting hotline, for example).
Obviously, officers posing as ordinary citizens, lying during proceedings and the entire concept of V-men, etc. is nothing new. However, I am deeply uncomfortable with a future in which LE and specific private actors just get a pass to deceive people with extremely convincing automated tech making this process easier and scaleable, and them having a path to create fake audiovisual material under the guise of "preventing crime", which is a rather vague and difficult to limit reason. Too much can be justified as being done for crime prevention, and it mostly hits people who are innocent or not convicted of a crime (yet), and also affects their friends and family members.
With the opening clause about law authorizations, member states could create authorizations allowing banks, fraud-monitoring firms, telecom providers, or platform operators to deploy undisclosed AI interactions or unlabeled synthetic content in quasi-law-enforcement settings just under the guise of detecting, preventing or investigating crime. The line between criminal investigation, compliance monitoring and fraud prevention is being blurred in a way that heavily puts us at a disadvantage. While the guidelines say that the authorization law must specify purposes, circumstances, and safeguards and respect the rights of third parties, there is not yet a definition of any minimum substantive safeguards, nor do they require independent judicial authorization every time. Most often, rights of third parties means things like copyright.
The mentioned exemptions, in my view, aid the creation of an environment of distrust online that the transparency requirement otherwise seeks to prevent. They circumvent safeguards against deception, impersonation and manipulation for the most powerful coercive actors we have! We require transparency because of risks to democratic processes and societal trust, but the exemptions remove those safeguards exactly where we are least able to contest or verify what is happening. It will become harder for defendants, journalists, oversight bodies, and other investigators to determine whether evidence, communications, or media were AI-generated or manipulated when LE AI meddled in it while unmarked and undisclosed. If a conversational AI used in an investigation hallucinates, misleads, escalates emotional pressure, or incorrectly infers intent, then that will that negatively and unfairly affect the outcome of the investigation.
At minimum, people should not unknowingly interact with highly persuasive synthetic systems capable of impersonation and emotional manipulation by (quasi-)policing actors. They deserve not having to constantly ask themselves whether something or someone they are interacting with is real, and possibly has LE manipulation behind it. The scale of deception the tech enables is intense, down to covert persuasion, emotional manipulation, or inducement, and we shouldn't just let cops (and wannabe-cops) have that unchallenged, with barely any oversight or limits.
I understand that for certain targets, transparency is ruining an investigation (child exploitation investigations, counterterrorism infiltration, etc.) but I expect this could increase risks of entrapment and manipulative practices, and an increase of chilling effects online as people adjust their behavior accordingly.
This should not be adopted like that without a lot of work addressing these issues and limiting the exceptions to specific cases.
Reply via email
Published
Providers are natural or legal persons, public authorities, agencies or other bodies that develop AI systems, or have them developed, and place them on the Union market (ex: OpenAI). Deployers are natural or legal persons, public authorities, agencies or other bodies using AI systems under their authority (ex.: universities that supply AI models to their students).↩
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。