Most AP automation is measured properly exactly once, during acceptance testing, and then never again. The project team disbands, the dashboard no one opens keeps updating, and eighteen months later the team has rebuilt manual workarounds for anything that has stopped working.

Degradation in document automation is undramatic, which is precisely why it survives so long.

Drift has a definition, and the model is rarely the culprit

The word drift makes people think of the model getting worse. In practice, the model is usually the last thing to change. What changes is everything around it.

A supplier moves to a new billing system and the layout shifts. Procurement adds a category and the coding rules no longer cover it. A country changes a tax rule. Someone tightens a blocking reason in the ERP, where invoices get blocked for payment when a price or quantity variance exceeds tolerance, and the exception queue grows for a reason that has nothing to do with document reading at all.

This is why the discipline that works is borrowed from risk management rather than from data science. The NIST AI risk framework asks organisations to monitor deployed systems continuously and to define in advance what a meaningful change looks like, in its Measure function playbook. Deciding afterwards what counts as degradation is how eighteen months pass.

The seven numbers to watch from week one

Pick these before go-live and put them on one page. Building the extract the first time is a week of work across several ledgers, especially if your reason codes are free text; once it's a scheduled job, the monthly review is an hour:

• First-time match rate. The share of invoices that match and post with nobody involved. When something drifts, this is the number that moves first.
• Exceptions per thousand invoices, split by reason code. The total tells you little. The split tells you everything.
• Postings the ERP rejects. The agent proposed, the ERP said no. A rising count means your rules and the ERP's have quietly stopped agreeing.
• Time to resolution per reason. An exception that waits four days and one that waits four hours are different problems with the same name.
• First-touch resolution rate. How often the first person to open an exception can close it without asking anyone.
• Proposal acceptance rate. How often a person accepts the coding as proposed, tracked by cost centre rather than in aggregate.
• Corrections that repeat. The same correction made twice in a month is a rule that needs updating, not a mistake.

None of these is an accuracy score, and that's deliberate. Accuracy is what the vendor measures. These seven are what your close depends on. The one exception worth making is the three fields where wrong money leaves the company: amount, bank identifier and VAT number deserve a per-field check of their own.

Every change goes on the same calendar as the numbers

The single practice that separates teams who catch drift from teams who discover it is boring: keep one calendar where every change is logged next to the numbers.

A tolerance widened, an instruction edited, a new supplier onboarded, an ERP patch, a country rule. When exceptions jump in March, the calendar tells you in a minute what changed in March. Without it, the investigation takes a fortnight and usually ends in a shrug.

Who owns the watching

This is an org-chart question, not one about tooling, and it's where most programmes fail. Project sponsors move on, vendors watch their own metrics, AP is measured on throughput, so no one is measured on whether the automation is still doing what it was designed to do.

Name a person and give them the one-page review, monthly, with the authority to open a change. A short standing meeting with an owner is what prevents the version of this story where nobody notices for a year and a half.

What the monthly conversation sounds like

It's short when it works. Exceptions are flat except for one reason code that doubled, which traces to a supplier that changed layouts on the ninth. Proposal acceptance dipped in one cost centre, which turns out to be a new category procurement added without telling anyone. Two corrections repeated, so two instructions get updated this week, recorded and dated.

That's the whole meeting. The numbers matter far less than the fact that somebody is looking at them every month.

How Hypatos makes AP report on itself

Hypatos puts this reporting inside the product instead of leaving it to a spreadsheet somebody maintains. The platform's insights give visibility into bottlenecks, processing time and straight-through rates, and the exception-handling interface carries the reasons and the suggested resolutions, so the monthly review reads from the same system that does the work. Whatever your provider does not report on, you track yourself from day one.

The instructions the invoice processing workforce follows are written in plain language, which is what lets the person who changes a rule also be the person who records it on the calendar. The agents propose, a named person decides, and the record shows what was checked, so the review can trace any number back to the invoices behind it.

Hypatos fits when you need the automation to report on its own health to a person who is accountable for it, instead of to a dashboard no one opens.

Start with last month

You don't need a project to begin. Pull last month's exceptions, split them by reason, and count how many were closed by the first person who touched them.

If you can't produce that split in an afternoon, you've found the first thing to fix.

‍