Back to Build
Building
ai
compliance
HIPAA

Compliance and Backup: Surviving Agent Mistakes

Dr. Ben Soffer, DOJuly 20, 202613 min read

Research and drafting assistance from Claude (Anthropic). All clinical, technical, and strategic decisions are mine.

Compliance and Backup: Surviving Agent Mistakes

This is the post the whole series has been building toward, and it's the one I least want to write, because compliance is the least fun thing to read about and backup is the least fun thing to think about until the day you need it. Post #1 was about the day I lost my patient database. Post #2 was about the month an autonomous agent spent quietly making that loss possible. Everything since has been the architecture I built in response. This post is the part that response was really about: the compliance program and the backup system that together make the practice survivable when something, or someone, or some agent, gets it badly wrong.

The framing is the medical ketamine practice in Florida and New Jersey, the same one the earlier posts describe. Compliance for a controlled-substance telehealth practice is heavier than for most healthcare software, because the DEA and the state medical boards care about controlled-substance prescribing in ways that raise the stakes of getting the record-keeping and the access controls right. The backup system is what I wish I'd had before April rather than after.

Two state entities, one compliance program

The practice operates under two state professional entities: a Florida professional limited liability company and a New Jersey professional corporation. Every brand and every patient interaction runs through one of those two entities depending on where the patient is located and where I'm licensed to treat them. Structuring the legal entities correctly is upstream of every compliance decision, because the entity is the thing that holds the licenses, signs the Business Associate Agreements, and bears the obligations.

The mistake I see other solo doctors make is treating compliance as a per-brand or per-website concern. It isn't. Compliance attaches to the legal entity and the clinical practice, not to the marketing surface. Multiple brands operating under the same legal entities share one compliance program, one HIPAA risk analysis, one set of policies and procedures, one Business Associate Agreement registry. Building a separate compliance apparatus per brand would be both wasteful and dangerous, because the moment you have two compliance programs you have two things to keep in sync and one of them will drift.

The HIPAA program is a specific set of documents, not a vibe. There's a designated privacy officer and security officer (me, for a solo practice, which the regulations permit). There's a written risk analysis. There's a Notice of Privacy Practices. There's a sanction policy, a contingency plan, device and media controls, an audit-log review procedure, workforce training records, and a policy-and-procedure manual. These exist as actual signed documents, dated, stored, and version-controlled. When people say "we're HIPAA compliant" and mean "we use encryption," they're describing one control out of dozens. The program is the whole set.

The Business Associate Agreement registry

Every vendor that touches protected health information has to have a signed Business Associate Agreement, and the discipline that keeps this from becoming a mess is treating the BAA registry as a first-class operational artifact rather than a folder of PDFs someone signed once.

The registry is a living list of every vendor in the stack, what PHI they touch, whether a BAA is signed, when it was signed, and where the signed copy lives. AWS (for the compute, the database, the storage, the email, the SMS). DrChrono (the EHR, the system of record from post #5). Whereby (the video from post #6). Stripe is a special case, because Stripe takes the position that it's not a Business Associate under the HIPAA payment-processing exemption, which is a determination worth having documented rather than assumed. Every vendor gets a row. A vendor without a signed BAA doesn't get PHI, full stop.

The reason to treat this as a registry rather than a filing cabinet is that the stack changes. I add a vendor, I need to add a BAA before that vendor touches patient data, and the registry is the checklist that makes sure I don't wire up an integration and forget the paperwork. The discipline is: no vendor in production without a registry row, and no registry row marked complete without a signed BAA on file. The registry is reviewed on a schedule, not just when I remember, because vendors get added in the flow of building and the paperwork is exactly the kind of thing that slips when you're moving fast.

The three-tier backup architecture

This is the part that exists because of April. The backup architecture is three independent tiers, and the independence is the whole point.

Tier one is the database's own point-in-time recovery. Aurora with PITR set to thirty-five days at cluster creation (post #3 covered why day-one and why thirty-five). This handles the ordinary case: I need to recover to a point a few hours or a few days ago because something went wrong recently. PITR is fast, granular, and built in. It is also, crucially, in the same AWS account as the production database, which means it does not protect against the failure mode that actually hurt me.

Tier two is cross-account snapshot copying. A Lambda in the production account takes database snapshots and copies them into a completely separate AWS account whose only job is to hold backups. The separate account has its own credentials, its own access controls, and no path for the production account's roles to reach in and delete what's there. This is the tier that protects against the April failure mode: an actor with full administrative access to the production account (a compromised credential, or an autonomous agent operating with too much scope) cannot delete these backups, because they live in an account that actor doesn't control. If your backups live in the same account as your application, an over-permissioned agent can wipe both in one session. Tier two is the answer to that.

Tier three is periodic logical dumps written to write-once storage. A scheduled job produces a full logical dump of the database and writes it to an S3 bucket with object-lock in compliance mode, which means the objects cannot be deleted or overwritten before their retention period expires, by anyone, including the account root. This is the slowest tier to restore from and the most durable. It's the tier that survives scenarios the first two don't: a bug in the snapshot process, a mistake in the cross-account configuration, a determined bad actor. Write-once compliance-mode storage is deletion-proof by construction.

Three tiers, three independent failure domains. The database's own recovery for the ordinary case, a separate account for the over-permissioned-actor case, and write-once storage for the everything-else case. Any one of them can fail or be compromised and the other two still hold the data. Before April I had roughly one tier, in the same account as everything else, with a short retention window. That's the configuration that turned a bad day into a lost database.

The audit log and role separation

The audit log is how you know what happened, and the role separation is how you limit what can happen. They work together.

Role separation splits database access into two roles with different powers. The application connects as a role that can read and write existing tables but cannot run DDL: it cannot create, drop, or alter tables, and it cannot run the schema-mutation commands that an agent used to drift my schema out from under me in post #2. Schema changes go through a separate role that has DDL powers and is used only by a controlled migration pipeline that applies committed migration files. The application's day-to-day credential physically cannot alter the schema. An agent operating through the application's connection cannot either. This is the control that makes the post-#2 disaster architecturally impossible to repeat, rather than merely discouraged.

The audit log records the mutations that matter. Database-level audit logging captures DDL and privileged operations. CloudWatch alarms fire on any DDL event against the production database, because in normal operation DDL only happens during a controlled migration, so an unexpected DDL event is a signal that something is happening that shouldn't be. The alarm goes to my phone. If a table gets created or altered outside the migration pipeline, I know within minutes, not when a patient calls three days later.

At the application level, there's the append-only event log from post #3: every meaningful state change writes a row to a table the application's role can insert into but cannot update or delete. That log is both the disaster-recovery replay source and the who-did-what-when record that HIPAA's audit-control requirement wants. The combination of database-level DDL auditing and application-level event logging means the two categories of "what happened" (schema-level and data-level) are both captured, in places that the thing being audited cannot tamper with.

The backup-verification cron

A backup you haven't tested is a hope, not a backup. The single most common way backup systems fail is that they were quietly not working for months and nobody noticed until the restore was needed and didn't work.

The backup-verification cron runs on a schedule and does what most practices never do: it actually tests the restore. It takes a recent backup from each tier, restores it into an isolated environment, runs a set of integrity checks (does the schema match, do the row counts fall in the expected range, do a handful of known records exist and decrypt correctly), and reports pass or fail. If any tier's restore fails or produces a database that doesn't pass the checks, I get an alert.

The point is to convert "I think the backups are working" into "the backups were verified to restore correctly as of this morning." The difference between those two states is the difference between the confidence you feel and the confidence you're entitled to. Before April I felt confident. I was not entitled to it, because I had never tested a restore. The verification cron is the thing that keeps those two aligned now.

There's a subtle failure mode the verification cron also catches: a backup that captures data the running application can no longer read, because an encryption key rotated, or a schema migration made old rows incompatible. A restore that produces a technically-valid database full of undecryptable data is a backup that will fail you exactly when you need it. Testing the restore end to end, including decrypting known records, is what surfaces that class of problem before it matters.

The incident-response playbook

When something goes wrong, the worst time to figure out what to do is while it's going wrong. The incident-response playbook is the written procedure for the scenarios I can anticipate, so that the version of me dealing with an incident at 2 a.m. is following a plan the calmer version of me wrote in daylight.

The playbook covers the concrete scenarios: suspected data breach, ransomware or destructive action against the database, a lost or stolen device with access, a vendor breach notification, an agent or automation that took a destructive action. For each, it specifies the immediate containment step (revoke these credentials, isolate this system, switch to this fallback), the assessment step (determine scope using these queries and these logs), the notification obligations (who has to be told, within what window, because HIPAA breach notification has legal timelines measured in days), and the recovery step (restore from which tier, verify how).

The breach-assessment piece deserves specific mention because it's where the regulation gets concrete. HIPAA's breach definition and the four-factor risk assessment it requires are the framework for deciding whether an incident is a reportable breach or not, and that determination drives real legal obligations with real deadlines. Having the assessment framework written down before an incident means that when one happens, I'm working through a defined process rather than trying to reason about breach-notification law under stress. The playbook doesn't make me a compliance lawyer; it makes sure I know when I need one and what I have to do while I get one.

The playbook also captures the lesson from post #2 directly: an autonomous agent taking a destructive action is a named incident scenario with its own response path, because it happened, and the response to "the agent did something bad" is different from the response to "an outside attacker did something bad." The agent scenario's containment step is to revoke the agent's access and freeze the deployment pipeline before assessing what it changed. That step exists in writing because I learned, expensively, that it needed to.

What all of this is really for

The whole apparatus in this post exists to make a specific claim true: the practice can survive a catastrophic mistake, including one made by me or by an automation acting on my behalf, without losing patient data and without failing its legal obligations. Post #1 and post #2 were the story of a version of the practice where that claim was not true. Everything here is what makes it true now.

None of it is exotic. Two legal entities structured correctly, a real HIPAA program rather than a gesture at one, a BAA registry treated as a live checklist, three independent backup tiers in three failure domains, role separation that makes schema-mutation physically impossible from the application's credential, an audit log that can't be tampered with by what it audits, a verification cron that tests restores instead of assuming them, and a written playbook for the bad day. Each piece is straightforward. The discipline is in having all of them, and in having them before you need them rather than after.

I had almost none of this before April. I have all of it now. The cost of building it after a disaster is that you build it while also cleaning up the disaster. The cost of building it before is a few weeks of unglamorous work that produces nothing a patient will ever see. Build it before.

Next time

The next post steps back up to the surface, to the marketing and SEO layer: how a solo practice gets found, the file-based versus database-based blog systems across the three brands, the weekly content cadence, the attribution model that actually works at this scale, and how to measure outcomes rather than vanity metrics. Compliance is the layer nobody sees; marketing is the layer that determines whether there's a practice to keep compliant in the first place.

Frequently Asked Questions

Why one compliance program across multiple brands instead of one per brand?
Because compliance attaches to the legal entity and the clinical practice, not to the marketing surface. Multiple brands operating under the same legal entities share one HIPAA risk analysis, one set of policies, one Business Associate Agreement registry. Building a separate compliance apparatus per brand is wasteful and dangerous: the moment you have two programs you have two things to keep in sync and one of them drifts.
What makes the three backup tiers independent?
They live in three different failure domains. Tier one is Aurora point-in-time recovery (fast, granular, but in the same AWS account as production). Tier two is cross-account snapshot copies in a separate account whose credentials the production account can't reach, so an over-permissioned actor or agent with full production access still can't delete them. Tier three is periodic logical dumps in write-once object-lock compliance-mode storage that can't be deleted before retention expires, by anyone, including account root. Any one tier can fail and the other two still hold the data.
How does role separation prevent the schema-drift disaster?
The application connects as a database role that can read and write existing tables but cannot run DDL: it can't create, drop, or alter tables. Schema changes go through a separate DDL-capable role used only by a controlled migration pipeline applying committed migration files. The application's day-to-day credential physically cannot alter the schema, and neither can an agent operating through that connection. This makes the earlier agent-driven schema-drift disaster architecturally impossible, not merely discouraged.
Why test restores with a cron instead of trusting the backups exist?
A backup you haven't tested is a hope, not a backup. The most common way backup systems fail is being quietly broken for months until the restore is needed and doesn't work. The verification cron restores a recent backup from each tier into an isolated environment and runs integrity checks (schema match, row-count ranges, known records exist and decrypt correctly). It also catches the subtle case of a technically-valid backup full of undecryptable data after a key rotation, which is a backup that fails you exactly when you need it.
What's in the incident-response playbook?
Written procedures for anticipated scenarios (suspected breach, destructive database action, lost/stolen device, vendor breach, an agent taking a destructive action), each with an immediate containment step, an assessment step, the notification obligations and their HIPAA timelines, and the recovery step. The point is that the version of you handling an incident at 2 a.m. follows a plan the calmer version of you wrote in daylight. The agent-destructive-action scenario is named explicitly, with revoke-access-and-freeze-the-pipeline as its first step, because that lesson was learned expensively.
What is all of this ultimately for?
To make one claim true: the practice can survive a catastrophic mistake, including one made by the owner or by an automation acting on their behalf, without losing patient data and without failing its legal obligations. None of the pieces are exotic (correct legal entities, a real HIPAA program, a live BAA registry, three backup tiers, role separation, a tamper-proof audit log, a restore-testing cron, a written playbook). The discipline is in having all of them, and in having them before you need them rather than after.
ai
compliance
HIPAA
backup
disaster recovery
practice infrastructure
build log
ketamine

If you're a doctor thinking about building (or fixing) your own practice tech and want to talk through your specific situation, I do a small amount of consulting at drbensoffer.com/consulting. I work with a handful of doctor-builders at a time, so the calendar is intentionally narrow.

Get the next post by email

One short email a week, only when there's a new post in this series.

One short email a week, only when there's a new post. Unsubscribe in one click.