AI, user data and the asymmetry of understanding

Every time users belatedly discover that an artificial intelligence feature has been drawing on their data in ways they did not fully grasp, the reaction is often an instinctive sense of violation – of trust, consent and privacy.

Accusations and outrage have always followed potentially invasive AI integrations, with examples ranging from email content used to inform model training and large on-device models embedded in everyday software to voice assistants retaining snippets beyond explicit commands and default settings that enable cross-product activity to inform AI responses.

Even when such changes are technically disclosed, awareness doesn’t necessarily follow. Updates arrive one after another, and settings default to “on,” putting the onus on users to navigate a labyrinth they never asked for. The cognitive gap between what organizations understand about how their systems use data and what individuals can reasonably expect to understand seems to widen daily.

Most users don’t mind disclosing chat, clickstream or location history if it serves their purpose, but companies on the other side may see training data, embeddings, personalization signals, safety-tuning inputs, fraud-detection features and future product capabilities in those messages.

Regulators are already acknowledging how upstream data decisions persist downstream. In late 2024, the European Data Protection Board updated its opinion on issues of anonymity, legitimate interest and AI models trained on unlawfully processed personal data, noting that this can affect whether such models can be lawfully deployed unless properly anonymized. The U.K.’s Information Commissioner’s Office also stresses the need for organizations to explain AI-assisted processes and decisions to those affected.

Burden on the user

 s it realistic to expect individuals to reverse-engineer opaque data ecosystems from privacy notices? Most people are simply trying to use products. To think about the downstream flow of their data, its implications and routes is overwhelming, to say the least.

In practice, the obligations should fall more heavily on companies. They design systems and understand their downstream uses. They are also the only actors positioned to reduce the complexity at the source. Meaningful transparency cannot be simply reduced to shorter privacy policies; it has to be contextual, specific and genuinely actionable.

This is not just a theoretical concern. Across the General Data Protection Regulation, ICO guidance and the EU AI Act, there is a recurring recognition that transparency must go beyond disclosure to become something people can actually understand and act on. They also push for explanations covering how data is used, who is responsible, and what consequences follow.

The EU AI Act is adding further transparency duties for certain AI systems, aimed at helping users recognize when they are interacting with AI or exposed to AI-generated content so that they can make informed decisions.

The catch in ‘manage your preferences’

Privacy responsibility is frequently redistributed toward users through interface design and rhetoric. The tendency to confer an impression of “control” through settings and toggles is likely to persist in one form or another, including dark patterns that may continue to lurk within interfaces. It’s a low-friction way for systems to signal compliance and user empowerment without actually changing the underlying distribution of power or reducing organizational discretion.

The primary responsibility should rest with those who shape the system’s architecture. Users should still have rights and controls, but companies are the ones deciding the defaults, retention periods, data flows, vendor relationships, and increasingly how models behave in practice.

This is ultimately a question of where responsibility is placed in systems that no longer follow simple, linear paths. If privacy risk is structural, it can’t be administered through settings and preferences alone. Accountability must be at the architectural level because that is where real decisions are made.

Onur Alp Soner is the co-founder and CEO of Countly Ltd, a digital analytics and in-app engagement platform. He wrote this article for SiliconANGLE.

Image: Wikimedia Commons