PHI in AI Prompts: How to Detect, Redact or Block Patient Data Before It Reaches a Model

PHI in AI prompts is patient health information that identifies someone, or could reasonably identify them, typed, pasted, uploaded or screenshotted into an AI tool. Control it in two moves: detect the identifiers, then decide by destination. A licensed tool covered by a BAA gets what the task needs. A personal account gets no patient data.

Redaction helps. It is not de-identification, and that distinction decides most of the policy. All patients and staff in our examples are fictional.

Last verified: 3 November 2026, against the text of 45 CFR 160.103, 164.402, 164.502 and 164.514 in the eCFR (current to 30 October 2026, read through the official eCFR API) and NIST IR 8053. HHS.gov guidance pages refused our automated requests, so no HHS guidance is quoted. The reliability ratings and default actions are our own assessment.

What counts as PHI under HIPAA?

Under 45 CFR 160.103, protected health information is individually identifiable health information held or sent in any form. It relates to a person's health, care or payment for care, and it either identifies the person or there is "a reasonable basis to believe the information can be used to identify the individual." A prompt with no name in it can still be PHI.

The famous list comes from de-identification. Under the Safe Harbor method in 45 CFR 164.514(b)(2), health information counts as de-identified only if these identifiers "of the individual or of relatives, employers, or household members of the individual" are removed. The wording is the regulation's own:

1. Names;
2. All geographic subdivisions smaller than a State, including street address, city, county, precinct, zip code, and their equivalent geocodes, except for the initial three digits of a zip code if, according to the current publicly available data from the Bureau of the Census: (1) The geographic unit formed by combining all zip codes with the same three initial digits contains more than 20,000 people; and (2) The initial three digits of a zip code for all such geographic units containing 20,000 or fewer people is changed to 000.
3. All elements of dates (except year) for dates directly related to an individual, including birth date, admission date, discharge date, date of death; and all ages over 89 and all elements of dates (including year) indicative of such age, except that such ages and elements may be aggregated into a single category of age 90 or older;
4. Telephone numbers;
5. Fax numbers;
6. Electronic mail addresses;
7. Social security numbers;
8. Medical record numbers;
9. Health plan beneficiary numbers;
10. Account numbers;
11. Certificate/license numbers;
12. Vehicle identifiers and serial numbers, including license plate numbers;
13. Device identifiers and serial numbers;
14. Web Universal Resource Locators (URLs);
15. Internet Protocol (IP) address numbers;
16. Biometric identifiers, including finger and voice prints;
17. Full face photographic images and any comparable images; and
18. Any other unique identifying number, characteristic, or code, except as permitted by paragraph (c) of this section

Removal is only half of Safe Harbor. The covered entity must also not "have actual knowledge that the information could be used alone or in combination with other information to identify an individual who is a subject of the information." The other route is Expert Determination, a documented finding by a qualified expert that the risk of identification is very small.

Where does PHI enter AI prompts?

Inside everyday work:

• Pasted notes and referral letters. Rachel Torres, a nurse, pastes a progress note for Adam Bailey, 71, and asks for a plain-language version for his daughter.
• EHR screenshots. A resident captures the medication list and asks about interactions. The identifiers are now pixels.
• Uploaded PDFs. A discharge summary to shorten, or a denied claim with a member ID, an account number and dates of service.
• Patient-list spreadsheets. David Kim in revenue cycle uploads 400 no-shows with names, phone numbers and appointment dates.
• Dictation transcripts. "She lives over on Maple, past the fire station" is an address with no structure at all.
• Prior-authorization and appeal letters. Sarah Mitchell pastes a denial letter, the member ID and three pages of history to draft an appeal.

A control that reads only the prompt box misses the images and files. Our post on whether DLP can catch uploads, screenshots and pastes maps which control layer sees which channel.

How does PHI detection work, and where does it fail?

Detectors layer four methods, and each has a known failure. A 2015 NIST review, NIST IR 8053, says rule-based systems for clinical text "tend to work well for specific kinds of text but may not work well when applied to new domains," while statistical systems generally perform less accurately but are easier to repurpose.

Pattern matching handles structured identifiers: SSNs, phone and fax numbers, email addresses, URLs, IP addresses and dates. NIST IR 8053 reports a University of Utah evaluation in which rule-based systems excelled at SSNs and phone numbers. Dates are the weak spot. A blood pressure of 120/80 and a dose of 5/325 both look like dates.

Dictionaries and context handle identifiers with no national format, such as medical record numbers, member IDs and device serials. The detector needs your formats and nearby words like "MRN". Untuned, it confuses an MRN with a lot number.

Named-entity recognition uses a model to tag people and places in free text. It is the only practical way to find names, and it fails both ways: Hope, Young and Rose are names and words. NIST IR 8053 adds that names "may not be clearly marked" and that medical terms can be mistaken for personal information, citing eponyms such as Bell's Palsy.

OCR turns screenshots and scans into text for the other methods. It struggles with handwriting, and it cannot tell that an image is a face.

One case defeats all four. "The nine-year-old from our county with a rare metabolic disorder, transferred last Tuesday" holds no name and no number, and a neighbor could still name the child. NIST IR 8053 puts it plainly: even after the 18 Safe Harbor elements are removed, "information may remain that allows identification of the medical subject."

Which identifiers can software catch, and what should happen to each?

"BAA tool" means a licensed AI tool covered by a business associate agreement for that product and plan, used with the work account. "Personal" means a personal or free account. Ratings and defaults are our assessment; tune them with your privacy officer.

The personal side is nearly all block, on purpose. The BAA side turns on what the task needs.

Should you redact, block or allow?

Three inputs, in this order:

1. Destination. No BAA for this product and plan, or a personal sign-in? Block clinical content, including files and screenshots. Redact stray identifiers, such as an email address, in prompts that are not about a patient.
2. Data type. Face photos, biometric files and bulk patient lists go only through an approved workflow, whatever the destination.
3. Minimum necessary. On a BAA tool, ask what the task needs. A family summary does not need the SSN or MRN, so redact them. An appeal may need the member ID, so allow it.

Why destination first? A vendor that "creates, receives, maintains, or transmits protected health information" on a covered entity's behalf is a business associate under 45 CFR 160.103, and 164.502(e) requires satisfactory assurances, documented in a written agreement, before PHI goes to one. A personal account has no such agreement with your organization, so sending PHI there, with no other Privacy Rule permission, is generally an impermissible disclosure. Which plans offer a BAA varies by vendor; see whether ChatGPT is HIPAA compliant.

The minimum necessary standard in 164.502(b) requires "reasonable efforts to limit protected health information to the minimum necessary to accomplish the intended purpose of the use, disclosure, or request." It has exceptions, including disclosures to a health care provider for treatment; ask your privacy officer how they apply to each AI workflow. And 164.514(d)(5) bars disclosing an entire medical record unless specifically justified, which a whole chart pasted for one paragraph rarely is.

If a disclosure is impermissible, 45 CFR 164.402 presumes a breach unless a risk assessment of at least four factors shows a low probability that the PHI has been compromised. The first factor covers "the types of identifiers and the likelihood of re-identification." A log of what was redacted is evidence for it. It does not decide the outcome.

Redact, Don't Block explains why blanket blocks push people to personal accounts. In healthcare, redact on the licensed tool and block on the personal one.

Why is redaction not de-identification?

Because HIPAA defines de-identification, and a prompt redactor does not meet that definition by default. Safe Harbor needs all 18 groups gone, including "any other unique identifying number, characteristic, or code," plus no actual knowledge. A redactor removes what it detects, and dates often stay because the clinical question depends on them.

The regulation even names the in-between state. A limited data set under 164.514(e) strips direct identifiers such as names and record numbers and is still, in the rule's words, "protected health information." Redaction reduces what is disclosed. That is valuable, and it is all it does.

What should you do now?

1. Inventory AI use with the account type. Each tool, department, and personal or work sign-in.
2. Decide the licensed destinations. Name the tools and plans under a BAA, and which surfaces of each are in scope.
3. Set defaults per identifier group. Start from the table and record the reasoning, which doubles as minimum necessary documentation.
4. Cover screenshots and uploads. Test EHR screenshots, scanned PDFs and spreadsheets, not just typed text.
5. Record violations as evidence. User, tool, account type, identifiers found, action taken.
6. Review false positives with clinicians. A monthly half hour finds blood pressures flagged as dates.
7. Train with real examples. Use the six scenarios above, with fictional patients.

The Healthcare AI Security Playbook helps at steps 2 and 5, because it puts the BAA review, the risk analysis and audit evidence into one rollout plan. For the wider HIPAA rules, read our HIPAA and AI guide.

What can software catch, and what can't it?

Software is strong on structured identifiers and weakest on the rare condition, the small-town detail and the face.

That is why the BAA and the destination matter more than the detector. On a licensed tool under a BAA, a missed identifier is a minimum necessary question. In front of a personal account, even a perfect detector still sends whatever it missed to a party with no obligations to you. Software can enforce the destination rule and keep the record; it cannot sign the BAA or run the risk analysis.

How does AccuroAI help?

AccuroAI redacts sensitive data inline, before the prompt is sent, with 40+ classifiers at under 38 ms p99. The classifiers cover personal data such as Social Security numbers, card numbers, account numbers and email addresses, and OCR reads text in screenshots. AccuroAI detects and redacts medical record numbers. MRN formats differ between health systems, so test detection against your own record formats during the pilot. Policies allow, redact or block per app, account type and data type, so personal accounts can be blocked while the licensed tool keeps working. Conversations are recorded only when a policy is violated, and audit logs stream to your SIEM.

Healthcare teams usually run AccuroAI on-prem or in their own cloud, so patient data stays inside their environment. For customers who choose AccuroAI-hosted deployment, AccuroAI signs a BAA. Whether the on-prem case calls for one is a question for your counsel. The healthcare page shows how the pieces fit.

FAQ

Is redacting a prompt the same as de-identifying it?

No. HIPAA de-identification means Safe Harbor, with all 18 identifier groups removed and no actual knowledge that the rest could identify someone, or a documented Expert Determination. A prompt redactor removes the identifiers it detects. What remains, such as dates or a rare condition, can still be PHI.

Can staff paste notes into a personal ChatGPT account if names are removed?

Not safely. Removing names leaves dates, places, record numbers and clinical detail, so the text is usually still PHI, and a personal account has no BAA with your organization. Block clinical content to personal accounts and give staff a licensed tool covered by a BAA.

Do EHR screenshots count as PHI?

Yes, when they show identifiable health information, which an EHR screen almost always does. The identifiers are pixels, so text-only rules miss them. Read images with OCR before they are sent, or block image uploads to destinations without a BAA.

Does the minimum necessary standard apply to AI prompts?

Generally, yes. 45 CFR 164.502(b) requires reasonable efforts to limit PHI to the minimum necessary for the purpose, which covers what goes into a prompt. Exceptions exist, including disclosures to a health care provider for treatment, so check each AI workflow with your privacy officer.

This post is security guidance for practitioners, not legal advice.

To learn where patient data already goes before you set any defaults, see it on your own devices: a 72-hour pilot maps the AI already in use before any policy is enforced. Start a free pilot

Sources

• 45 CFR 160.103, Definitions, eCFR
• 45 CFR 164.402, Definitions (Breach Notification Rule), eCFR
• 45 CFR 164.502, Uses and disclosures of protected health information: General rules, eCFR
• 45 CFR 164.514, Other requirements relating to uses and disclosures of protected health information, eCFR
• NIST IR 8053, De-Identification of Personal Information, NIST, October 2015