The last two posts in this series are about meta-tools: the systems I built not to run the practice directly, but to make my decisions and my agents better at running it. This one is about a pattern I call the Conclave, which is what I use when a decision is important enough that I don't trust a single model's answer to it, including my own.

The setup is simple to describe and took a while to appreciate the value of. Instead of asking one language model for its recommendation on a hard question and acting on the answer, you ask several models independently, have them critique each other's reasoning, and then synthesize from the disagreement. The name is a joke about cardinals deliberating behind closed doors, but the mechanism is serious, and it's changed how I make the decisions that are expensive to get wrong.

I didn't invent this. The Conclave came to me from a friend in tech named Ben Lamm, along with a related tool called FAMP that I'll cover in the final post. Ben writes Operator Error, a newsletter of field notes from someone who actually builds with AI tools instead of theorizing about them: half experiments, half "this is what broke today." Worth saying his name out loud, because I owe him. Solo doctors building production infrastructure are not as alone as they used to be, if they know which tools exist. His newsletter is at https://operatorerror.substack.com/.

What follows is how I've adapted the Conclave for the decisions that come up running the practice, and the one case where it rescued me from a problem I genuinely could not solve any other way.

The single-model trap

Here's the failure mode the Conclave exists to prevent. You ask a capable language model a hard question. It gives you a confident, well-written, plausible answer. The answer is wrong in a way you can't see, because the model's confidence and its correctness are only loosely correlated, and a fluent wrong answer is indistinguishable from a fluent right one if you don't already know the right one.

This is fine for low-stakes questions, where being wrong is cheap and you'll find out fast. It's dangerous for the decisions that are expensive and slow to reveal their wrongness: an architecture choice you'll build on for a year, a positioning decision for a brand, a clinical-operations policy, a vendor commitment. On those, a single model's confident answer is a trap, because the fluency reads as reliability and there's no second opinion in the room to puncture it.

There's a second, subtler version of the trap that matters even more, and I'll get to it through the case below: a single model is excellent at proposing the next fix within an approach, and structurally bad at noticing that the approach itself is wrong. It will help you optimize your way deeper into a dead end, one plausible step at a time, and never suggest turning around.

What the Conclave actually is

The Conclave is a structured way to capture the disagreement between models. The mechanism has three phases.

First, independent answers. The same carefully-framed question goes to several different models, each without seeing the others' responses. Independence matters here; if they see each other's answers first, they anchor, and you get consensus that's really just contagion. Each model produces its own recommendation and its reasoning.

Second, cross-critique. Each model is shown the others' answers and asked to critique them: where is this reasoning weak, what did it miss, what would make it fail. This is where the value concentrates, because a model is often much better at finding the flaw in another model's argument than at avoiding the flaw in its own. The blind spot that a model can't see in itself is frequently obvious to a different model with different training.

Third, synthesis, which is the Conclave's job and not mine. The Conclave has a judge that reads the full record of independent answers plus critiques and produces the final recommendation, weighing the strongest surviving reasoning against which critiques landed and which were themselves rebutted. It isn't a vote count. My role isn't to write that synthesis; it's to interrogate it. I read the judge's recommendation and send it back with more questions and more detail, it comes back sharper, and I send it back again, until I'm satisfied the reasoning actually holds. The output I act on has survived both adversarial review by peer models and my own iterated questioning, which is why I trust it more than any single model's first confident answer: not because it's a consensus, but because it's been pressure-tested from both sides.

The whole thing is a few model calls and some orchestration. It's not expensive in compute or time relative to the cost of getting an important decision wrong. And it's the same principle as convening a panel of human experts who don't all think alike, except that it's available in minutes and the experts don't have egos about being critiqued.

A prompt template that works

The framing of the question matters more than anything else in the Conclave, because a badly-framed question produces confident answers to the wrong problem from every model at once, and cross-critique won't save you if they're all answering the wrong thing.

The template I've settled into has a few required parts. State the actual decision, not a proxy for it. Give the real constraints, including the ones that are embarrassing or specific to my situation, because a model reasoning without the real constraints produces a textbook answer that doesn't survive contact with the actual problem. Ask explicitly for the reasoning and the failure modes, not just the recommendation, because the reasoning is what the other models will critique and the failure modes are what I actually need to weigh. And require a concrete recommendation rather than a survey of options, because "it depends" from every model is useless; I want each one to commit to an answer I can then stress-test.

The critique phase gets its own framing: show each model the others' committed recommendations and ask specifically what would make each one fail, not whether it's good. "What would make this fail" produces sharper critique than "what do you think of this," because the former demands a specific mechanism and the latter invites polite hedging.

The synthesis phase is the judge's: given these recommendations and these critiques, which reasoning survives, and what's the decision. Not which model won. Which reasoning survives. Then I interrogate that answer, sending it back with more questions and detail until it holds up, which is the part that actually takes the time.

A real case: the video visit that would not work

The clearest example of the Conclave earning its keep was the video visit, which was the single hardest technical problem in the entire build. I mentioned earlier in the series that it was where I spent more debugging time per line of code than anywhere else. This is the rest of that story.

The requirement was easy to state: a patient should be able to join a HIPAA-compliant video visit from inside the patient portal, on the same domain, without bouncing out to a separate app or typing their name onto a stranger's landing page. Easy to state, and for a long stretch I simply could not get it to work. The embed failed in maddeningly inconsistent ways. It worked in one browser and not another. It worked in development and broke in production. It collapsed under the portal's strict security policy into blank frames and cryptic console errors that pointed at nothing actionable.

I worked the problem with a single model, Claude, the way I worked everything then: describe the failure, get a fix, try it, describe the new failure, get the next fix. We went around that loop many times. Every iteration was locally reasonable. Adjust this security-policy directive. Try that embed parameter. Restructure the frame this way. And every iteration moved the failure somewhere new without ever resolving it. This is the single-model trap in its purest form. The model was genuinely good at proposing the next fix within the approach, and it never once stepped back to ask whether the approach was the problem. Neither did I, because each fix was plausible enough to keep me believing the next one would be the one that landed. Numerous iterations in, I had a portal that still couldn't reliably do a video visit and a growing pile of changes that each felt like progress and summed to none.

Eventually I convened the Conclave, and this time I framed the question correctly. Not "why won't this embed work," the proxy question I'd been grinding on for weeks, but "how should video visits work in this portal, given these HIPAA and same-domain constraints," which is the actual decision. That reframing was the whole difference. Freed from debugging the broken approach, the panel questioned the approach itself instead of continuing to service it. One model recommended, specifically and with its reasoning laid out, switching to Whereby Embedded, whose embed model was built for exactly this same-domain authenticated-portal case and whose HIPAA-compliant configuration was a documented, supported path rather than something I was fighting the framework to force into existence. The critique phase stress-tested the recommendation and the other models agreed the reframe was right. The synthesis was unambiguous: stop trying to make the current approach work, switch.

It worked. The thing I had spent an embarrassing amount of time failing to force into place with a single model came together quickly once the Conclave changed the question from "fix this" to "is this even the right this." Post #6 describes the Whereby Embedded integration that resulted; what that post doesn't mention is that I only arrived at Whereby because the Conclave broke a loop that Claude alone and I had been stuck in for far too long.

The lesson generalizes well past video. A single model is superb at proposing the next step within a framing and structurally poor at noticing the framing is wrong, because questioning the framing isn't the next step, it's a different kind of move. When you've iterated on something many times and it still doesn't work, the problem is usually the approach rather than the execution, and that is exactly the realization a single model in a fix-this-error loop is least likely to reach and a well-framed panel is most likely to surface. The Conclave didn't out-debug Claude. It did something Claude in a debugging loop couldn't do for itself: it asked whether we were debugging the right thing.

Six guardrails

The Conclave has failure modes of its own, and these are the guardrails I've learned to apply.

First, frame the question yourself, carefully, and don't outsource the framing to a model. A model-framed question smuggles in the framing model's assumptions, and then the panel debates within those assumptions instead of questioning them. The framing is the human's job, and as the video case shows, the framing is often the entire game.

Second, insist on real independence in the first phase. If the models see each other early, you get anchoring dressed up as agreement. Independence is what makes the disagreement informative.

Third, weight critiques by whether they survive rebuttal, not by how many models raised them. A sharp objection from one model that nobody can rebut is worth more than a vague concern three models echoed. Consensus is not the goal; surviving reasoning is.

Fourth, watch for correlated blind spots. Models trained on overlapping data share some errors, so unanimous agreement is weaker evidence than it feels, because they might all be wrong the same way. The cases where every model agrees confidently are exactly the cases to stay a little suspicious of, because a shared training-data blind spot produces exactly that signature.

Fifth, don't accept the judge's first synthesis as final. The Conclave produces a recommendation; my job is to interrogate it, sending it back with sharper questions and more detail until the reasoning actually holds, rather than taking the first well-written answer because it's well-written. The Conclave is a decision-support tool, not a decision-maker. The point is to inform my judgment with structured disagreement and then pressure-test the result, not to hand the decision to whatever the judge returns first. I make the call; the Conclave makes the call better-informed.

Sixth, reserve it for decisions that deserve it. The Conclave is overhead, and using it for low-stakes questions is a waste that also dulls your sense of when it matters. I use it for the expensive-to-reverse and the stuck-for-too-long, and nothing else. Most questions get a single model and move on; the Conclave is for the few that are worth convening one.

Why this generalizes

The Conclave is a specific instance of a general principle that runs through this whole series: on the decisions that matter most, a single confident source is a liability, and structured disagreement is how you get to a reliable answer. It's the same principle as the three-tier backup architecture (post #8), where independent failure domains protect against any single one being compromised, and the same principle as the multi-channel safety alerting (post #5), where no single notification channel is trusted alone.

For a solo operator, this principle is load-bearing in a way it isn't for a team, because a solo operator doesn't have colleagues to disagree with in the hallway. The Conclave is a way to manufacture the productive disagreement that a team gets for free and a solo practitioner otherwise goes without. It's the closest thing I have to a second senior person in the room on the decisions where I most want one, and on the video visit it was the difference between a problem I couldn't solve and one I solved in an afternoon once the question was framed right.

Next time

The final post in the series is about the coordination layer that lets the fleet of agents from post #10 actually work together: the messaging protocol they use to hand each other context, the failure modes of multi-agent coordination, and why the thing that makes a fleet useful is the same thing that makes it dangerous if you get the boundaries wrong. The Conclave is how I make better decisions; the coordination layer is how the agents make better decisions together. Both are meta-tools, and both came after the practice was already running, because you don't know you need them until you've felt the absence.