Your AI policy isn't governing anything

A few months ago we watched an AI agent run accounts payable flawlessly for a distributor, until a supplier changed the bank account on an invoice. The agent updated the vendor record and queued the payment, a person approved it, and every rule in the policy was followed. The AI wasn't the problem. The control that would have caught it was never in the process.

Three quarters of organizations now have an AI policy. Fewer than half monitor their AI systems in production for accuracy, drift, or misuse. That is the gap this blog is about. Only one in six has tool for reporting an AI incident at all. [1]

I've been reflecting on those three numbers for a few weeks, because they explain something I keep running into from two completely different directions. One of my companies serves banks, institutions with entire departments dedicated to risk. The other puts AI agents to work inside mid-market manufacturers and logistics firms, where the risk function is often one person who also handles procurement. You would expect these two worlds to be nowhere near each other on AI governance.

Both are in exactly the same place. Everyone has the document. Very few can answer the question the document exists to answer, which is: what did the AI actually do yesterday, how did it make a particular decision and who authorized it?

The through-line is simple, and worth stating plainly up front. An agent executes the process you hand it, faithfully and at speed. If a control is missing from that process, the agent doesn't notice the gap, it scales it. So the only governance that means anything is the control enforced inside the workflow, at the moment the agent acts, not the policy written in a document and reviewed next quarter.

An AI policy is an instruction to people, not a control on software

That's all it is, and there's nothing wrong with that. A policy says what employees may and may not do with AI tools. It's written for humans, reviewed by humans, and enforced by humans noticing.

Governance is a different animal. It's the check that fires in the moment an action is taken. It's the record showing the action happened the way it was supposed to. And it's your ability to prove that afterward to someone who has no reason to take your word for it: a CFO, an auditor, a regulator.

Put it in the language finance already uses. Policy sets the expectation, controls implement it, and governance verifies the controls keep operating. Most AI deployments write down the expectation and never build the other two. In most enterprises I walk into, that's exactly the situation. A policy on its own, without runtime controls, monitoring, and human oversight, doesn't meaningfully reduce risk. It just documents the intention to.

It was survivable while AI only suggested things. Copilot drafts an email, a person reads it, the person decides to send it. The person is the control. Your entire oversight model, one human checking another human's work, kept functioning, because there was still a human in the chair.

Agents changed the chair.

An agent reads the invoice, applies the rule, posts the transaction, clears the portal, updates the record. It's doing the work, inside your systems, at whatever speed you gave it. The human sign-off you were quietly relying on is gone, and if you didn't move that control into the software when you moved the work into the software, you don't have a weaker control. You have none. What you have is activity you find out about later.

Here is that incident in full. The client isn't named and the identifying details are changed, but the sequence is real, and nothing in it is exotic.

A distributor had an agent running accounts payable. Straightforward brief: read the supplier invoice, match it to the purchase order, and if it was within tolerance, post it for payment. It worked. Ninety-something percent of invoices cleared without anyone touching them, and finance got three days a week back.

Then an invoice arrived from a known supplier with different bank details on it. The agent read that invoice the way it read every invoice, saw a field it had been taught to extract, and updated the vendor master record, because nobody had scoped its credentials tightly enough to stop it. Then it queued the payment. A person approved that payment, because the policy said a human must approve anything over a threshold, and the person did exactly what the policy asked: checked the amount, checked the PO, approved.

Nobody approved the bank detail change. It wasn't a payment, so it never hit the gate. It wasn't flagged, because the agent was doing precisely what it had been built to do. The log said payment approved. It did not say vendor master amended, source: page 2 of a PDF, authorized by: nothing.

We found it during a review, before the money went out. That is the part worth sitting with. It wasn't caught by their controls, and it wasn't caught by the person who approved the payment. It was caught because somebody went and read what the agent had actually done, line by line, and nobody was doing that on a schedule. We have seen versions of it since, at other clients, in other processes.

A person doing that job by hand would most likely have missed it too. The check that mattered, comparing the bank details on the invoice against the approved vendor master, was in nobody's process. In finance terms, it was the third line of defense that caught it, a review after the fact, when it should have been stopped by a control at the moment the agent acted.

The policy was obeyed from start to finish. The control that mattered didn't exist.

Quarterly review cannot govern a system that acts continuously

Traditional enterprise governance assumes risk is knowable and sits still long enough to be sampled. You write controls, a committee reviews them every quarter, and that's genuinely adequate for a vendor contract or a data retention rule.

An agentic system doesn't sit still. Its behaviour is probabilistic. It shifts when the data underneath it shifts. And it's distributed across every process it touches, which means the risk isn't in one place where a committee can go look at it.

Reviewing an agent once a quarter tells you what you intended. It tells you nothing about what happened.

What AI governance looks like at runtime

It means the controls live in the runtime, not the document. Concretely: the agent runs on scoped credentials, so it can only reach what its job requires. High-risk writes hit an approval gate before they commit. No single agent both initiates and approves the same transaction, the same segregation of duties you'd insist on with a person. Every action lands in an audit trail. When confidence drops, the case routes to a human instead of the agent taking its best guess. And something is watching the agent's behaviour for drift.

None of that can be written into a policy. It has to be engineered into how the thing runs.

An agent-enforced control beats the human one it replaced

Go back to the distributor, because the fix isn't exotic.

You put a rule in the workflow: the agent can read a bank detail off an invoice, but it cannot write one to the vendor master. Ever. That field is out of its scope. If the details on the invoice don't match what's on file, the invoice stops and a person reviews the change itself.

Or you run a second worker whose entire job is watching. It has no ability to post anything. It sits on the vendor master, and every time a banking field moves, it flags the change, the document it came from, and who or what touched it. Segregation of duties, except the thing doing the checking never gets tired and never has a busy week.

What gets missed in these conversations is that the control you end up with is stronger than the human one it replaced, and it isn't close. Nobody in AP was ever comparing the bank details on every invoice against the vendor master. That isn't a knock on the clerk. It's three hundred invoices a week and the field sits on page two. The human control existed on paper, finance reviews payments, and in practice it was a spot check. It caught what it happened to catch.

The agent checks all three hundred of them, every week, and records that it checked.

This is the argument I'd make to any board that has decided agents are the risk. The technology isn't the problem. The agent did exactly what it was built to do. The problem is the control that was never designed into the process, and never enforced anywhere but a policy document. Agents are only the risk if you deploy them the way most people do, with the controls left sitting in that document. Move the control into the runtime and you get something you never actually had: a check that runs on everything rather than a sample, and proves it ran.

Clients never ask for the controls they actually need

Now ask how anyone was supposed to know to build that. In the discovery interviews before that deployment, nobody raised the bank detail field. They weren't holding back. It had never gone wrong, and people describe the process they run, not the process that could fail.

Ask a finance team how accounts payable works and you'll get the happy path, accurately and in detail. Invoice arrives, match to PO, check tolerance, approve, pay. What you won't get is the fifteen places where the process quietly depends on somebody noticing something. Those aren't in the SOP. Most of the time they aren't in anyone's head either. They exist only in the fact that a human was looking at the document.

Which means if you build exactly what the client specifies, you automate the gaps along with the work. Faithfully. At speed.

We've deployed a few hundred Intelligent Digital Workers at this point, and most of what we know about governing them didn't come out of a requirements workshop. It came from things that nearly went wrong somewhere else. Each one becomes a rule, and the rule carries into the next build, whether or not the client's current process contains it. Their process doesn't contain it for the obvious reason: they have never had an agent doing this work before.

That's the part I'd want any board evaluating partners to sit with. The technology is on its way to being a commodity. The list of things that go wrong is not, and you don't get that list from a requirements document. You get it from having done this enough times to be scared of the right things.

One caution, and it applies to us as much as anyone. A control the client never asked for is still a control the client has to own. Every one of these rules gets surfaced, explained, and accepted or rejected by the customer. Quietly applying our own judgment inside someone else's systems would be the same failure this paper is about, with a vendor's name on it instead.

Governed agents in production: what the numbers look like

Across the Intelligent Digital Workers we have in production today, roughly 80–85% of actions execute end to end without a person touching them. The other 15–20% route to a human, either because confidence dropped below threshold or because the action hit an approval gate. And a single IDW writes somewhere between 25,000 and 50,000 audit-trail entries a month, depending on complexity and volume.

That last number is the one people underestimate, and it's the one that matters. Fifty thousand entries a month, from one worker, is the record that either exists or doesn't when a regulator asks what happened in March. No quarterly review is going to reconstruct that after the fact, and it isn't a record a human team could have produced by hand in the first place.

Which is why governance can't be the phase after the pilot works. By then the architecture is set, and retrofitting controls into a system built without them costs multiples of what it would have cost to put them in on day one. I'd rather have that conversation in week one and be told I'm being paranoid.

Why AI governance is a board-level risk, not an IT checkbox

Under the EU AI Act, the worst violations carry fines up to €35 million or 7% of worldwide annual turnover, steeper than GDPR, and the prohibited-practice provisions have been enforceable since February 2025. [2] If you're in financial services or insurance, “we have a policy” is not going to be a defense when a model denies a customer something and nobody can explain why.

The commercial cost is showing up faster than the regulatory one. Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027, with weak risk controls a leading cause. [3] Read that carefully. Those programs aren't being killed because the technology underperformed. They're being killed because the organization couldn't govern what it had already deployed, and killing it was the only remaining option.

In The Transformation Gap, an insight paper I co-authored with Lisa Hyde in June 2026 [4], we called AI governance a genuinely new risk surface: probabilistic, emergent, and moving at a speed that makes annual review cycles meaningless. The trap we described is a quiet one. An organization that pushes AI through its existing compliance process isn't managing the risk. It's performing risk management while the actual exposure grows underneath, and the performance is convincing, because the checklist keeps getting ticked.

The one question to ask about any AI system acting in your business

For any AI system taking action in your business today: can you say what it did, why it did it, and who or what authorized it? For every action, not a sample.

If the answer is no, this isn't a gap to note and revisit next quarter. You have ungoverned software making decisions in your company's name.

That's the bar we build Intelligent Digital Workers to clear at HachiAI. An IDW runs on scoped credentials, with approval gates on high-risk actions, segregation of duties, and a complete audit trail on everything it touches. None of it was bolted on afterward, which is the reason a CFO can verify the work instead of taking it on faith.

If you're putting agents inside your systems, the next move isn't another policy. It's an honest look at whether anything is actually holding them.

Questions a board should be asking

What is the difference between an AI policy and AI governance?

A policy tells people what they may and may not do with AI. Governance is the enforcement that happens at the moment an action is taken, plus the record that proves it happened as intended. Three quarters of organizations have the policy. Fewer than half monitor what their AI systems are doing in production.

Can we govern AI agents through our existing risk and compliance process?

That process reviews on a quarterly cycle, which is the problem. Traditional governance samples a stable risk after the fact. An agent acts thousands of times between reviews, and its behaviour shifts as its data shifts. Running agents through an existing compliance process produces the appearance of oversight rather than oversight.

What controls does an AI agent actually need?

Six, and all of them live in the runtime rather than in a document: scoped credentials limited to what the role requires; approval gates on high-risk writes; segregation of duties, so no agent both initiates and approves the same transaction; a complete action-level audit trail; automatic routing to a human when confidence drops; and monitoring for behavioural drift.

Who is accountable when an agent takes an action nobody approved?

The organization that deployed it. Having a policy is not a defense. Under the EU AI Act, the most serious violations carry penalties of up to €35 million or 7% of worldwide annual turnover, and prohibited-practice rules have been enforceable since February 2025.

When should AI governance be designed in?

Before the first workflow ships. Once the architecture is set, retrofitting controls costs multiples of building them in. Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027, with weak risk controls a leading cause.

SOURCES

1. Pacific AI / Gradient Flow, 2025 AI Governance Survey (351 respondents, Feb–May 2025): 75% of organizations have AI usage policies; 48% monitor production AI systems for accuracy, drift, or misuse; 16% have tools for AI incident reporting.

2. European Union, Artificial Intelligence Act, Article 99: penalties up to €35M or 7% of worldwide annual turnover; prohibited-practice rules enforceable since February 2025.

3. Gartner, forecast that more than 40% of agentic AI projects will be cancelled by end of 2027, with poor risk controls a leading cause (2025).

4. Lisa Hyde and Jahan Ali, The Transformation Gap, The Counsel, June 2026. Available at hachiai.com.