Common Ways AI Can Lead to HIPAA Violations

The impact of AI is being felt across the healthcare industry, enabling organizations to streamline operations, enhance patient engagement, and improve clinical outcomes. However, AI use in highly regulated industries like healthcare comes with elevated risks, specifically HIPAA violations that can lead to costly fines.

From chatbots and transcription tools to predictive analytics, AI is becoming embedded in day-to-day healthcare functions in a variety of ways. Each one of these uses can expose Protected Health Information (PHI), creating potential for HIPAA violations. This article will break down the most common ways AI can lead to HIPAA violations and how organizations can mitigate those risks.

AI-Powered Chatbots

AI-powered chatbots are now commonly used to interact with patients, assist with appointment scheduling, answer basic medical questions, handle prescription refill requests, and even conduct symptom triage. While these tools are efficient, they also create significant privacy concerns. Patients may disclose PHI (medical records, prescriptions, personal identifiers, etc.)  to the chatbot. That data could be exposed if not properly secured. Chatbots are often deployed through third-party vendors. In many cases, those vendors may not be HIPAA-compliant.

Violation Scenario

A patient shares symptoms and personal details via a chatbot. The AI platform logs the conversation, including patient names, stated symptoms, and appointment details, then stores that data without encryption in an unsecured environment.

Best Practices for Compliance:

• Execute a signed Business Associate Agreement (BAA) with every chatbot vendor before any PHI flows through the system.
• Encrypt data in transit and at rest.
• Clearly disclose chatbot usage in your Privacy Notices.
• Conduct a thorough review of the vendor’s data retention, logging, and model training policies. Ensure no patient data is used to train AI models without a separate HIPAA-compliant authorization framework.
• Implement automatic session timeouts and prohibit browser-based caching of PHI in chatbot interfaces, particularly on shared or public devices.
• Audit chatbot access logs quarterly and include chatbot-related data flows in your annual HIPAA Risk Assessment.

Not Conducting Risk Assessments Before Implementing AI

HIPAA requires covered entities and business associates to conduct and document a thorough risk assessment when new systems that handle PHI are introduced. New AI systems are no exception, as they often create novel attack surfaces, data flows that are difficult to follow, and include complex vendor relationships. Many organizations, in a hurry to leverage AI, may fail to perform a proper risk assessment, leaving vulnerabilities in their data collection, storage, and sharing practices.

The absence of a documented risk assessment is a HIPAA violation, regardless of whether a breach occurs.

Best Compliance Practices

• Establish a formal AI governance policy requiring a HIPAA security risk assessment before any AI tool that touches PHI is deployed.
• Map all PHI data flows associated with the AI system, inputs, outputs, training data, logs, and API calls, as part of the risk assessment documentation.
• Include the AI vendor’s security architecture, penetration testing results, and compliance with security frameworks in your risk analysis documentation.
• Integrate AI systems into your organization’s broader risk management framework and make updates whenever a new AI tool is introduced.

Using Patient Data to Train AI Models Without Authorization

For healthcare organizations and their vendors, using patient data to train AI models creates complex HIPAA compliance challenges. Under HIPAA, PHI can only be used to train AI models if strict conditions are met, such as obtaining patient authorization, ensuring data is de-identified, and having a BAA.

Using identifiable information to train models creates both privacy and legal risks. Simply removing a patient’s name from a clinical note does not satisfy HIPAA’s de-identification requirements. All eighteen categories of identifying information specified in the Privacy Rule must be removed, and expert determination must confirm that the remaining data carries no reasonable re-identification risk.

When working with a third party, Data Use Agreements (DUAs) should clearly define how vendors can use data. Organizations should prohibit secondary uses without approval.

Violation scenario:

A health network partners with an AI radiology vendor, giving it imaging studies with only patient names removed to improve its model. The studies contain date-of-birth, geographic indicators, and rare disease markers sufficient for re-identification. The data was not properly de-identified, and no patient authorization was obtained, constituting unauthorized use of PHI for commercial benefit.

Best Compliance Practices

• Conduct an overall analysis before permitting any patient data to be used for AI model training, clearly determining whether the use constitutes permissible healthcare operations or requires patient authorization.
• De-identification must go beyond names. Rigorously apply all 18 Safe Harbor identifiers before using data for training.
• Data use agreements (DUAs) must lay out clear data use rules for vendors.
• If patient data will be used to train models that benefit a commercial vendor beyond your own operations, obtain proper patient authorization under HIPAA’s authorization requirements.
• Establish a data governance committee with authority to approve or reject AI training data use cases before vendor agreements are signed.
• Audit vendor contracts annually to identify any data-use provisions that may have been added in updates that may conflict with your HIPAA obligations.

Using AI for Transcription

AI transcription tools allow healthcare providers to convert conversations into structured medical notes. The efficiency benefits are clear, but audio data containing PHI will often be included, meaning these recordings must be stored, transmitted, or processed in a HIPAA-compliant manner. Transcription tools may upload recordings to public cloud environments, retain data indefinitely, or even use it to train AI models. To mitigate these risks, healthcare organizations should use only HIPAA-compliant transcription vendors willing to sign a BAA.

Aside from HIPAA concerns, many state laws, such as CIPA, require all-party consent for audio recording.

Violation scenario:

A doctor’s office deploys a transcription tool without informing patients that conversations are being recorded. The vendor’s system stores audio files in an unencrypted cloud environment. A misconfiguration makes the data publicly accessible, creating HIPAA breach notification obligations and potential wiretapping liability through laws like CIPA.

Best Compliance Practices

• Ensure BAAs with transcription vendors specifically address audio storage, retention periods, deletion schedules, and prohibitions on using audio or transcripts for model training without separate authorization.
• Ensure encryption in transit and at rest for audio and transcripts.
• Implement a clear patient notification and opt-out process before any ambient recording begins. Determine whether affirmative consent, opposed to just a notice, is required in your jurisdiction.
• Require that all audio files be automatically deleted within a defined window after transcript generation is complete. Verify this deletion occurs through audit logging.

Misconfigured AI Cloud Services

Many AI applications rely on cloud infrastructure, which introduces another layer of complex data security. Misconfigured cloud environments are a common cause of data breaches. Simple configuration errors, such as leaving a storage bucket publicly accessible or failing to restrict API access, can expose large volumes of PHI.

The shared responsibility model of cloud computing can be confusing. Cloud providers maintain the security of the infrastructure, while covered entities and business associates are responsible for configuring that infrastructure securely. Many organizations mistakenly believe that using a HIPAA-eligible cloud service automatically makes their deployment compliant, but that’s not the case. HIPAA-eligible cloud services provide services that can be configured to meet HIPAA security standards for storing, processing, or transmitting PHI. A BAA with a cloud provider means the provider will help you meet HIPAA requirements. It does not mean your configuration is compliant by default.

Violation scenario:

A hospital’s AI vendor deploys a predictive readmission model on cloud environment. Model inference logs, including patient identifiers and clinical inputs used as model features, are stored in a public bucket with anonymous public access enabled, resulting in the logs being indexed by external web crawlers.

Best Compliance Practices

• Require that all cloud resources used for AI systems touching PHI be inventoried and put through a rigorous security audit.
• Apply the principle of least privilege to all AI service accounts, API keys, and role assignments. No AI system should have broader data access than its specific function requires.
• Enable cloud-native audit logging for all storage resources and AI services that process PHI, and route logs to a SIEM for anomaly detection.
• Prohibit public access configurations on any storage bucket, container, or database that could contain PHI.
• Include vendor cloud configuration standards in your BAA requirements and conduct annual technical audits of vendor cloud environments that process PHI.

HHS provided this guidance on HIPAA and cloud computing.

Entering PHI into Public AI Tools

The growth of publicly available AI platforms creates a new HIPAA risk that healthcare organizations must account for: employees entering PHI to streamline their tasks. Tools like ChatGPT, Copilot, and Gemini are widely accessible and easy to use, making them an appealing option for summarizing patient notes, drafting communications, and other projects. However, most free versions of public AI tools are not designed to meet HIPAA requirements and do not offer BAAs. PHI entered into public AI platforms could be stored, logged, or even used to train AI models, creating an immediate violation.

Violation scenarios:

• A physician puts patient names, diagnoses, and medication lists into a public AI assistant to draft consult notes.
• A billing specialist asks an AI tool to help code a complex encounter by describing the patient’s diagnoses and procedures in detail.
• A nurse pastes a patient’s clinical history into a public AI chatbot to generate a discharge summary.

Best Compliance Practices

• Develop and publish a clear, written AI use policy that explicitly prohibits the entry of PHI into any AI tool that does not have a signed BAA with your organization. When implementing AI tools, commit to using enterprise-grade, HIPAA-compliant solutions.
• Deliver mandatory HIPAA and AI training to all clinical and administrative staff, specifically illustrating what constitutes PHI in the AI context and providing concrete examples of prohibited and permitted uses.
• Implement technical controls where feasible, including DLP (Data Loss Prevention) tools that can detect and block PHI from being pasted into non-approved web applications.
• Establish an amnesty reporting mechanism that encourages staff to self-report accidental disclosures, enabling the organization to conduct prompt breach risk assessments rather than discovering incidents later.

The power of AI is allowing healthcare organizations to operate more efficiently and better care for patients, but it must be managed carefully to avoid violating HIPAA. At CompliancePoint, we specialize in HIPAA compliance, and we have expertise in AI risk management. Put our knowledge to work for your organization, so it can harness the power of AI responsibly and maintain compliance. Reach out to us at connect@compliancepoint.com to learn more about our services.