The rise of AI-powered health tools is prompting new thinking about how, where, and when sensitive health information receives legal protection. According to media reports, consumers are now using general-purpose AI tools to upload or query health information, including medical records, and several companies have recently released large language model (LLM)-based tools customized for consumer health uses. While such records are protected by the Health Insurance Portability and Accountability Act’s (HIPAA) Privacy Rule when collected by healthcare providers and health plans, they largely fall outside HIPAA’s protections once uploaded to consumer-facing AI platforms.

Using online tools to seek health information is not new and consumers have long used health and wellness wearables and apps to share medical information for holistic health experiences, often with beneficial outcomes. Where downloading medical records is still a frustrating or limited experience, policy and technical architectures have  emerged to facilitate consumer-directed health information-seeking. What is new is the underlying health data architecture utilized by AI tools and LLMs. This new architecture is a combination of policy shifts, product features, and public privacy commitments – setting new consumer expectations for how consumer health data should be handled based on old frameworks like HIPAA.

This blog post examines the emerging architecture of AI-powered health tools and its implications for privacy, governance, and consumer protection. We explore:

• A New(ish) Health Data Architecture: Under mandatory patient access policies, patients are transferring HIPAA-protected data out of covered environments—effectively stripping it of its HIPAA status—where it is commingled with consumer health information in AI and LLM-based tools. Novel data and privacy protection practices and policies are emerging that seek to meet patient-consumer expectations for protection and may establish new industry standards around health data architecture.
• Key Implications: This evolving architecture raises critical questions about regulatory applicability when medical records move outside HIPAA’s scope, how platforms should handle inferred health information about non-users, whether AI can adequately account for clinical nuance and medical judgment, and how to measure the effectiveness of voluntary privacy safeguards. 

Traditional Health Data Architectures: Client-Server

A fundamental challenge in consumer health technology has been navigating the governance practices and technical architecture needed to handle two categories of health information regulated by distinct legal frameworks. Medical Records or Protected Health Information (PHI) held by healthcare providers, health plans, and their business associates are protected by  HIPAA—a highly regulated, entity-based framework that attaches protections based on who holds the data and in what context it was collected. Consumer Health Data or Information, by contrast, collected by commercial entities like health and wellness apps that fall outside HIPAA’s scope, is governed by a variety of state consumer privacy and protection laws—which are data-based frameworks where protections depend on the type of information collected and where the individual lives. This divide is not new: even before AI, online symptom checkers and wellness tools required personal health information to function while operating outside HIPAA’s regulatory perimeter.

Until recently, patient portals which siloed HIPAA-protected data in authorized environments and consumer informational websites used a similar technical architecture of client-server. In a client-server architecture, users would input information into a web-based form (client), which would then send this data to a central server that would store the data according to protection requirements. Servers would run a pre-programmed, rules-based logic engine to organize, analyze, and respond to user requests or queries. The process was largely deterministic and relied on the explicit technical rules encoded by human experts. 

Patient/Consumers as the New(ish) Arbiters of Data and Privacy

Another architectural shift in policy is the enforcement of the individual patient’s power to access and move their electronic health records (EHRs) between systems and protection frameworks. Individuals have historically had tacit but inconsistent access to their electronic health information (EHI) as facilitated by the HIPAA covered entity. Federal law penalizes Information blocking where medical information is not accessible to individuals who are entitled to access. The 21st Century Cures Act (Cures Act) defines information blocking as “a practice by an individual or entity that is likely to interfere with, prevent, or materially discourage the access, exchange, or use of electronic health information except as required by law or as specified in an information blocking exception.” As of February 2026, the Information Blocking Complaint Portal has been open and actively used, with over 1,600 complaints submitted and some predicting enforcement in the future.

The requirement for HIPAA-covered entities to facilitate access and transfer of EHI, per the Cures Act, allows individuals to control that version of their information and upload or transfer it as they want or need. This policy transformation is facilitated by an associated technical shift under the Cures Act where healthcare entities must maintain standardized APIs allowing data to be interoperationalized and more easily moved between systems. This interoperability and access is the essential precursor to the consumer health AI that may include previously HIPAA-protected information. Without this first step, many individuals may not have had easy access or transferability of EHI, whereas now, individuals may largely access, download, and upload their EHI at will with few barriers. Simply put, individuals are, now more formally, the arbiters of their own data and privacy protections regarding their EHI and will choose which systems to move their medical records into or out of. Individuals, however, may or may not be aware of what protections apply, meaning that at least in regards to data and privacy protections, individuals may not always be making informed decisions when moving their EHI. 

LLMs can integrate patient-accessed and -uploaded medical records with non-HIPAA consumer health data; these systems go far beyond existing querying tools familiar to stakeholders and into longitudinal, pattern-aware platforms. This convergence creates a centralized point of sensitive and variably-regulated health data, fundamentally shifting privacy obligations and trade-offs for all stakeholders throughout the data lifecycle.

The New Architecture for Health Data Protection 

The confluence of shifting from deterministic client-server to AI architectures and the evolution of individuals’ access to the information in their medical records changes health data systems. This technological and regulatory evolution redefines how organizations handle consumer health data, creating new data ecosystem practices and expectations.

Key examples of the practices starting to emerge in response to this evolved ecosystem center on public promises to maintain HIPAA level data and privacy protections for consumer data. Simultaneously, governance frameworks beyond traditional regulations—such as voluntary public commitments and AI ethics boards—proactively manage AI risks. These technical and policy changes, coupled with heightened privacy commitments that exceed legal requirements, establish new expectations for handling consumer health data (regardless of HIPAA status). This new architecture—involving technology, policy, and design— merits careful evaluation, introducing challenges in explainability, bias, and control that require innovative policy and technical responses.

Examples of Revised Architectural Approaches

Some entities have explored mechanisms for revising this traditional architecture. For example:  Health Data Segmentation and Expanded Protection Promises: While traditional consumer health tools may have had the option to upload health records from various sources, the health records would not remain separate or receive additional protections. Once a user had voluntarily shared health data, regardless of source, with a non-HIPAA entity, the data was protected in the same way as other health or wellness data. Some AI companies are now purporting to implement “purpose-built isolation, separate memories, and compartmentalized storage” – continuing the practice of allowing individuals to upload their medical records but offering separate digital space for centralization that also encompassed health and wellness data.  Data Minimization, Necessity Requirements, and AI Training Policies: Another growing piece of policy architecture emerges around how AI platforms and downstream entities handle user data, which can be understood through two distinct regulatory developments with potentially overlapping impacts. First, new laws and regulations are increasingly imposing substantive data minimization requirements that tie the collection or processing of personal data strictly to what is “necessary” to provide a requested product or service. If these necessity requirements are interpreted narrowly, or if they fail to include exceptions for routine activities such as product improvement and development, they may effectively prohibit companies from training AI models on uploaded or shared consumer health data, regardless of company’s promises to not train on the data. Second, distinct from general data minimization rules, new AI-specific laws and regulations may take a more direct approach by outright banning the training of AI models on some or all user input data. Together, these legal frameworks aim to limit the potential for secondary uses and unintentional data leakage, fundamentally shifting the responsibility for data protection upstream to foundational AI providers. This proactive and multi-pronged approach underscores that restricting data use is likely an essential aspect of data governance as consumer health data increasingly intersects with powerful, continuously learning AI systems.

Critical Implications This Architecture Raises

Regulatory Fragmentation Meets Novel Architecture in Health Data and Privacy Protections 

The architectural convergence of patient-controlled and interoperable HIPAA-protected data with non-HIPAA health data creates unique regulatory compliance challenges. When individuals upload medical records to AI platforms that aren’t HIPAA-covered entities, that data may lose HIPAA protection and become subject to a fragmented patchwork of state and federal laws—with protections varying significantly based on the user’s location and the nature of the data. 

What makes this particularly complex for LLM-based health tools is that the same system may be simultaneously subject to multiple, sometimes conflicting, regulatory frameworks. A single platform might need to comply with:

• FTC prohibitions on unfair and deceptive practices and the Health Breach Notification Rule
• State restrictions on AI use in mental health contexts (e.g., Illinois’ Wellness and Oversight for Psychological Resources Act)
• State health privacy laws that exceed HIPAA protections (e.g., California’s Confidentiality of Medical Information Act)
• Comprehensive state privacy laws and health-focused privacy laws (e.g., Washington’s My Health My Data Act and Virginia SB 754)
• Genetic privacy laws when records contain genomic information (e.g., Texas HB 2545 and the Texas Genomic Act) 
• Youth protection laws, when known minors share health information that contains or signals their age and with verifiable-parental consent (VPC.)

The bottom line: both standard general purpose LLMs and health-focused LLMs will be subject to similar standards of consumer protection, privacy, and AI laws. Furthermore, where companies publicly state a health-focused LLM will have increased protections due to the sensitive nature of the health information uploaded to the LLM, regulators may enforce those public statements.

Multi-Party Consent for Auxiliary Data in Single-User Systems

Though the conversation around consumer health AI often remains focused exclusively on the data of the individual user who is sharing, medical records and health conversations often contain information about people who didn’t consent to share their data with an AI platform. Although a platform may not retain the medical record itself, a range of information and inferences may be drawn from the information it contains. This auxiliary data can include:

• Group Data: Information pertaining to third parties, such as children, older adults, or family members, whose data may be present in a shared record (including separately regulated genetic information), regardless of the user’s purpose for uploading it (which could range from well-intentioned care coordination to malicious use).
• Provider Data: Identifiers, tax numbers, notes, or references concerning healthcare providers and potentially medical facility staff, raising privacy concerns for these employees.
• Intellectual Property: Data that may be subject to copyright, trade protections, or clinical secrecy (e.g., specific internal protocols, proprietary clinical methodologies, or copyrighted educational materials found within a medical record).

Because traditional consent frameworks often assume a single data subject, this architecture reveals the limitations of that assumption.

​​Clinical Judgment Meets Algorithmic Interpretation

Medical practice routinely involves judgment calls that fall outside standard protocols but serve patients well. Off-label prescribing (e.g. using FDA-approved drugs for conditions they weren’t officially approved to treat) is one common example. This practice is evidence-based and widespread in clinical medicine, but general-purpose LLMs may flag it as incorrect or potentially dangerous, creating questions around liability.

The implication extends beyond off-label prescribing to any clinical decision involving nuance: evolving treatment guidelines, patient-specific contraindications, or the expert reasoning that experienced practitioners apply to complex cases. When AI systems interpret these decisions as errors rather than judgment calls, they risk undermining the patient-provider relationship and creating confusion about appropriate treatment. The challenge is designing systems that can acknowledge uncertainty and defer to clinical expertise rather than treating medicine as a domain with algorithmic certainty.

Conclusion

The integration of AI into health data introduces a new challenge by centralizing highly-regulated medical records with less-regulated consumer health information, often outside of HIPAA protections. This shift raises critical questions about the practical implementation of technical privacy safeguards, the management of sensitive “auxiliary data” (like information about family members or providers) within uploaded records, and the ability of AI models to interpret complex clinical nuances, such as off-label prescribing. Moving forward, clarity in protections and applicable state and federal regulations are crucial to ensure the benefits of these changing technologies going forward.