Exception Queue Design for Human-in-the-Loop
Exception queue design for human-in-the-loop workflows: the honest straight-through processing rate, who staffs the queue, and how to size it before you build.
Vous voulez le framework derrière ces projets ?
Obtenez le système Claude Code que nous utilisons pour planifier, construire, tester et livrer des logiciels en production.
Problem: You bought automation to run a process end to end. It runs most of it. The leftovers land in a list somewhere, nobody owns that list, and three months later the list is the process.
Quick Win: Design the leftovers before you design the automation. An exception queue is the list of items the software stopped on and handed back to a person. That handback is what "human in the loop" means: a named person finishing specific work at a specific point, not a vague promise that someone is watching. And the leftovers are permanent. In accounts payable, the team that pays supplier bills, the average company processes just 32.6% of invoices with no human touch at all, and even the top 20% of performers only reach 49.2% (Ardent Partners, 2025). The queue is not a gap in the product. The queue is the product.
What An Exception Queue Actually Is
Automation is a set of rules and models that finish work without asking. An exception is any item those rules refuse to finish: the software stops, flags it, puts it in a list. Say an invoice arrives with no purchase order number on it, the number that ties a bill back to what the company actually ordered. The checking step cannot do its job, so the invoice goes in the list. Someone finds the order, and the invoice carries on.
Two definitions worth keeping straight, because vendors blur them:
- Straight-through processing rate (sometimes called the touchless rate): the share of items that go from arrival to done with zero human involvement.
- Exception rate: the share of items that get actively flagged and blocked.
Those are different numbers. Ardent's 2025 accounts payable study reports 32.6% fully hands-off alongside a 14% exception rate. So two-thirds of invoices get touched by a person somewhere, but only 14% are actually stuck. A lot of human involvement is not exceptions at all. It is work the automation never covered in the first place.
When somebody sells you 90% automation, ask which of those two numbers they mean. The gap between them is where the real staffing cost hides.
The Honest Target: Nobody Hits 100%
Here is the number that should end most automation debates. In the accounts payable benchmark above, invoice exceptions are now the single biggest challenge in the industry, named by 53% of teams, the first time that has happened in nineteen years of running the study. And only 5% of accounts payable teams say they have fully automated the handling of those exceptions, with another 60% calling it partly automated (Ardent Partners, 2025).
Read that honestly. One of the most heavily automated finance processes in business, running for two decades on mature software, hands roughly half its volume back to a person on purpose, and almost nobody has automated the handback itself.
Rates vary enormously by process. Anyone quoting one universal number is selling something:
| Process | What "no human touch" means | Published rate |
|---|---|---|
| Supplier invoices | Bill received, checked, approved and paid with nobody touching it | 32.6% across all companies. 49.2% for the top 20% of performers, 23.4% for everyone else (Ardent Partners) |
| Health insurance claims | Claim approved or denied automatically by rules | Roughly 80% cited as the industry benchmark, with real rates observed from 10% to 70% (HCIM, a claims-software vendor) |
| Car insurance claims | Claim settled without a human assessor | Up to 80% on car claims, reported by one vendor about its own customers (Assured) |
Structured, rules-heavy work automates high. Judgment-heavy work does not. Your target comes from your own process, not from a case study about someone else's. And note that the last two rows are vendors quoting their own numbers, which is exactly the kind of figure to check before you plan around it.
APQC, an independent benchmarking body, comes at the same process from a different angle. Its open-standards measures for accounts payable are cost per invoice processed, the time from receiving an invoice to sending payment, and the share of payments that are error free the first time (APQC). None of those reward a high automation rate on its own. They measure whether the whole thing, queue included, finishes cheaply and on time.
Three Kinds Of Exception, And Only Two Of Them Shrink
This is the framework that changes how you build. Sort every exception into one of three buckets:
| Type | Example | Who clears it | Does it shrink over time? |
|---|---|---|---|
| Missing data | Invoice with no order number. Lead with no company name. | Queue owner, or whoever created the gap earlier in the process | Yes. Fix the form, the supplier setup, the step where the data comes in |
| Unclear data | Two possible customer matches. A total that could be before or after tax. | Queue owner, using judgment | Partly. Better rules and better matching cut it, never to zero |
| Judgment call | A discount above the limit. A refund over a set amount. A non-standard contract term. | A person with authority to say yes | No, and it should not. This is the work you deliberately kept |
Most projects fail because they treat all three as bugs. They are not. The third bucket is the design. If your rules say a human approves discounts above 20%, every one of those is the system working, not a defect.
So the target is not "get the queue to zero." It is: drive buckets one and two toward zero, and let the queue settle on a stable floor of judgment calls. A queue that stops shrinking at 6% of volume, all of it judgment calls, is finished. A queue stuck at 20% with half of it missing data is broken further up the line, which is the argument for cleaning the data before you automate anything.
Who Reads The Queue, And What Their Day Looks Like
One named person. A stated response time. A named backup. That is the whole staffing model, and skipping it is the most common way these projects die.
Their day: open the queue twice at fixed times, work items oldest first, and record a short reason label for every single one. Not "resolved." A reason. No order number. Duplicate supplier record. Discount above the limit. Those labels are what turn the queue from a chore into a feedback loop.
They also need one power most builds forget to give them: the ability to overrule the machine and stop it. Europe's AI Act writes this into law for high-risk AI systems. Whoever oversees the system must be able to "disregard, override or reverse the output" and to "interrupt the system through a 'stop' button or a similar procedure," and must stay aware of automation bias, the human tendency to over-rely on whatever the system recommends (EU AI Act, Article 14). Whether or not that law applies to you, the design is right. A reviewer who can only click approve is not oversight. It is decoration.
The Aging Rule: What Happens At Hour Four, Day One, Day Three
A queue without an aging rule is a pile. Write this down before you build:
| Age | What must happen | Why |
|---|---|---|
| 0 to 4 hours | Owner sorts every new item and tags it with a reason. Sorting, not solving. | Sorting is fast, solving is slow. Never let the two block each other. |
| End of day 1 | Item is cleared, or handed to a named person outside the queue with a due date. | An item with no name attached is one nobody will finish. |
| Day 3 | Automatic escalation to that person's manager. The supplier or customer is told there is a delay. | Day three is when they find out on their own and lose trust. |
| Day 7 | It stops being an exception and becomes an incident, with a written cause. | Seven-day items are never one-offs. They are a broken rule in disguise. |
None of these rules are about how fast things get resolved. They are about how fast things get owned. Slow answers are survivable. Unowned items are not.
Feed The Queue Back So It Gets Smaller
Every item in the queue is a free bug report about your process. Most companies throw them away.
The loop: each month, count the reason labels, take the top three, and fix them at the source rather than in the queue. If most of your exceptions are "no order number," the fix is not a faster reviewer, it is how you set up new suppliers. If most are duplicate records, the fix is at data entry. In chasing money customers owe you, if most are payments that cannot be matched to an invoice, the fix is the payment detail your customers send along with the money.
The payoff shows up in the same benchmarks. Top performers in accounts payable run a 9.0% exception rate against 22.0% for everyone else, and process an invoice in 3.1 days for $2.78 while everyone else takes 17.4 days at $12.88 (Ardent Partners). The same software is available to both groups. The difference is whether anyone acts on what the queue says.
Where This Goes Wrong
A queue nobody owns becomes a second inbox. This is the default outcome. Security teams already ran this experiment at scale: a survey of nearly 300 security leaders and practitioners, run by a security software vendor, put the typical team at about 960 alerts a day, with roughly 40% never investigated at all (Prophet Security). Nobody decided to ignore 40% of the alerts. The volume decided for them. Your exception queue does the same thing, quietly, at a smaller scale.
The reviewer starts rubber-stamping. Give a person a long list of machine recommendations and they stop checking. This is well documented: over-reliance on automation shows up in beginners and experts alike, gets worse when the reviewer is juggling other tasks, and cannot be overcome with simple practice (Parasuraman and Manzey, Human Factors, 2010). The fixes are structural, not motivational: keep the queue small enough to read, re-check a sample of cleared items, and never make approve the fastest path.
One queue for everything. A blocked $400,000 invoice and a misspelled company name do not belong in the same list. Split by consequence, not by which system produced the item.
Measuring the automation rate instead of how fast the queue clears. "We are at 78% automated" tells you nothing if the other 22% takes eleven days. Always report two numbers together: the hands-off rate and the median age of items sitting in the queue.
Exceptions logged as errors. If the software files judgment calls in the same bucket as failures, your dashboard shows a system that looks broken and you waste a quarter chasing it. Tag the three types separately from day one. It is the specific version of a broader pattern in why most AI automation projects fail: nobody defined what success was allowed to look like.
How To Size The Queue Before You Build Anything
Do this arithmetic on a napkin before you sign anything. The method is real. The inputs below are illustrative, so use your own.
- Monthly volume. Say 2,000 invoices.
- Expected exception rate. Use a published benchmark until you have your own: 14% average, 9% for top performers, 22% for the weakest (Ardent Partners).
- Multiply. At 14%: 280 items a month, about 13 per working day.
- Minutes per item. Time yourself clearing ten real ones. Say 6 minutes.
- The answer. 13 items at 6 minutes is roughly 80 minutes a day. At the weak-performer rate of 22% and 8 minutes an item, closer to 2.5 hours a day, most of a half-time role.
Now you know whether this is a corner of somebody's job or a person you have to hire. Run that number before the build, not after launch. It is the same discipline as knowing your quote turnaround time or where approvals stall in a deal desk, the group that signs off on pricing and contract exceptions. Sizing the human half is what separates automation that survives from automation quietly abandoned in month four.
Related Reading
- Why most AI automation projects fail, the failure numbers behind the queue nobody staffed
- Automating the money customers owe you, where exception queues carry real cash consequences
- Deal desk automation, moving approvals from a weekly meeting into a queue with an aging rule
Frequently Asked Questions
What hands-off rate should we aim for?
Whatever your process type supports, and never 100%. In accounts payable it tops out near 49% for the best 20% of companies (Ardent Partners). For health insurance claims, one vendor cites roughly 80% as the benchmark while noting real rates run from 10% to 70% (HCIM). The variable is how much judgment the work needs, not how good the software is.
Should the person reviewing the queue be senior or junior?
Split it. Missing-data and unclear-data items are junior work, handled by whoever sits closest to where the data came from. Judgment calls need someone with real authority to say yes, so the queue has to route to two different people. Sending everything to one senior person turns the queue into a bottleneck instead of a safety net.
Can AI clear the exception queue too?
Some of it. Missing data and unclear matches are exactly where a model can propose an answer for a person to confirm. Judgment calls are not. Be careful here: a second layer of AI clearing the first layer's leftovers is where over-reliance does the most damage, because the reviewer now has two machines agreeing with each other and one tired person checking (Parasuraman and Manzey, 2010).
What is the single metric to watch?
Median age of open items. Not queue size, not automation rate. A queue of 300 items with a median age of four hours is healthy. A queue of 12 items with a median age of nine days means nobody owns it.
Most automation gets sold on the part that works. We build the other half too: the queue, the reason labels, the aging rules, the named owner, so the leftovers stay a small, shrinking, well-run list instead of an inbox nobody opens. That is what department automation that holds up looks like, and the same discipline goes into every process we install inside a company.
Vous voulez le framework derrière ces projets ?
Obtenez le système Claude Code que nous utilisons pour planifier, construire, tester et livrer des logiciels en production.