The Steps to Getting AI Right
How an AI system earns its way into your operation, one step at a time.
The question everyone opens with is will it work.
It’s the right question and it has no honest answer in advance. Nobody — not me, nor a vendor with a case study — can tell you in a first meeting whether a system will work in your operation, because most of what determines that isn’t the software. It’s how your files are actually filled in, which of your people will use it, and whether the thing it’s good at is a thing you needed.
As such, I structure the work so the answer arrives in pieces, early, while the cost of being wrong is still small. The mechanism is simple, and once you’ve seen it you’ll want it from any vendor: the system starts with no authority and earns more, one step at a time, on evidence.
Four steps. Most good deployments spend a long time on the first two, and plenty of them never go further — not from a failure to finish, but because that’s the right place for that operation to land.
Step 1: Shadow
The system runs. It reads your data, applies its rules, produces its output — and nobody acts on any of it. Either nobody sees it, or two or three people see it and treat it as a curiosity.
This sounds like a waste of a month. It’s the most valuable month in the project, for a reason nobody expects going in: shadow mode is where you find out what your data actually looks like.
Every operation I’ve worked with believed its records were in better shape than they were. Not through negligence — through the ordinary drift of a busy place. Dates entered in three formats. A status that means one thing to the person who set it up and something else to the person using it now. A field everyone abandoned two years ago that the reporting still keys off. You don’t discover this by asking. You discover it when software tries to reason over it and produces something obviously wrong, and someone says “oh — that’s because we stopped using that field.”
That discovery is worth the month by itself, and you’d have paid for it eventually anyway, in worse circumstances.
Shadow mode also gives you the only comparison that matters. The system says these eleven files need attention. What actually happened to those eleven files? Which ones did your people already catch? Which did the system miss? Which did it flag that turned out to be nothing? You now have a real number instead of a vendor’s number, from your operation, on your data.
Move up when: its output stops surprising you in bad ways, and someone on your staff says “actually, that one’s a good catch.”
Step 2: Advisory
Now it tells people things, and people decide what to do. Nothing is automated. The system has no ability to change anything.
The metric here is not accuracy. It’s whether anyone acts on it.
This is the step where most deployments quietly die, and the failure is almost never that the software was wrong. It’s that the output landed somewhere nobody goes, or arrived in a form that took more work to act on than to ignore. A dashboard nobody opens is a dashboard nobody opens, no matter how good the analysis is.
So this step is mostly a design problem, not an AI problem. Does it appear where people already work, or does it require a new habit? Can someone act on a flag in one step, or do they have to go find the file themselves? Is it showing them the fifteen things that are fine along with the two that aren’t?
That last one matters more than it sounds. A system that shows you everything is a system you learn to skim. The useful version shows you the exceptions — the file that went quiet, the assessment that’s overdue, the document that should be there and isn’t — and stays silent about the rest. Silence is a feature. It’s what makes the alerts mean something.
Move up when: people are acting on it without being reminded to, and they complain when it’s down. That complaint is the real acceptance test.
Step 3: Assisted
Now the system drafts and a person approves, and the approval is recorded.
This is the first step where anything the software produced touches your actual records, and it touches them through a human being who clicked something. That distinction is the entire safety property. It also means you now have a record of who approved what, which turns out to be the most valuable byproduct of the whole arrangement — more on that below.
The design question at this step is what “approve” costs. If reviewing a draft takes as long as writing it, you’ve built something worse than nothing, and people will route around it within a month. The draft has to be right often enough that approving is genuinely faster, and the interface has to make the wrong ones obvious rather than making all of them look equally finished.
A good tell: after a few weeks, ask someone how often they edit before approving. If it’s almost never, the review has become a rubber stamp and you’ve effectively skipped to step 4 without deciding to. If it’s almost always, the drafting isn’t good enough yet. Somewhere in between is where the step is doing its job.
Move up when: you can point at a specific, narrow action where the approval has been a formality for months, and where being wrong wouldn’t hurt much.
Step 4: Acting
Some things happen without a person.
Note the word some. The mistake is treating this as a switch you throw for the system. It isn’t — it’s earned per action, and most actions never earn it, and that’s the correct outcome rather than a failure to finish.
Attaching a received document to the checklist item it obviously satisfies: fine. Marking a routine task complete when its completion is unambiguous from the data: fine. Changing a deadline, altering a resident’s care level, sending anything to a client or a family member: no. Not because the model can’t, but because the cost of being wrong is asymmetric and no efficiency gain justifies it.
The tempting rule is to let the system act when it’s confident. Resist that one. Confidence is highest exactly where a mistake is most plausible, and it lets the software promote its own work. Tie the step to what the action can damage, which doesn’t change, rather than to how sure the model is, which changes constantly and isn’t observable by you.
Three things that make the steps work
A record you can compare against. The steps run on evidence, and evidence needs a log. From day one, the system should be recording what it saw, what it flagged, who looked, who acted, and what was denied to whom. That log is what lets you say “it’s been right about this for four months” instead of “it feels like it’s working.”
One person who owns it. Not a committee, and not me. Someone inside your operation who notices when the flags stop making sense and has the standing to say so. Deployments without this person don’t fail dramatically; they just slowly stop being used, and nobody can say when it started.
An agreed definition of wrong. Before step 1 ends, you and I should be able to state what a false alarm is and what a miss is, in your terms. It sounds pedantic. It’s the difference between a project that can improve and one where everybody has a vague feeling.
Picking the first problem
Not “AI for the firm.” One specific thing. The good candidates share four traits:
- It already happens repeatedly. A once-a-quarter problem gives you no signal for a year.
- Somebody already does it by hand. Then you have a baseline, and a person who knows what right looks like.
- You can tell afterward whether it was done correctly. If success is unmeasurable, the steps have nothing to build on.
- It doesn’t have to be right the first time. Which is exactly what shadow mode is for.
The best first problems are usually the ones already costing you quietly. Files that go dormant and nobody notices until it matters. Checklists that are complete in principle. Documentation gaps you find out about when someone external comes to look. These are unglamorous, they’re expensive, and they’re solvable without asking anyone to change how they work.
The worst first problem is the one that demos well.
What the first ninety days actually look like
Roughly: two to three weeks getting data flowing and normalized, which is longer than anyone wants and is where most of the real engineering is. Then three or four weeks of shadow, mostly spent finding out what your records look like. Then advisory, with the design changing more than you’d expect once real people see it — the first version of any flag is always slightly wrong in a way nobody could have predicted from a meeting.
Ninety days in, a good outcome is: it runs on your data, it surfaces things your people didn’t catch, they use it without being asked, and there’s a log that shows all of that. Not autonomy. Not headcount reduction. A working instrument and evidence about whether to build the next thing.
That’s a modest promise, and I’d rather make it and keep it.
The part that pays off later
Here’s the argument I’d make even to someone who doesn’t want AI at all.
Everything the steps require — the log of what was seen, who acted, who approved, what was denied — is the same record you need on the day somebody asks you to prove how you operate. A surveyor. Opposing counsel. A family member with a reasonable question. An insurer.
Most operations reconstruct that story afterward, out of email and memory, under time pressure. A system built this way produces it continuously as a side effect of running, because the record isn’t a compliance chore bolted on at the end — it’s the same mechanism that lets the software earn trust in the first place.
That’s the part I find genuinely satisfying to build. The careful version isn’t the slow version that eventually catches up to the reckless one. It’s the version that leaves you with something you can show.
I’m not a lawyer and I’m not a compliance consultant. I build software, and I design it against the rules that apply to the business it’s for. What your obligations are is between you and your own counsel — I can only tell you what I designed for and why.