Agent Review Inbox
An AI agent reviews a company's card spend. I designed the inbox where a finance manager handles what the agent isn't sure about, and teaches it when it's wrong.
61 sec · Before and after, narrated in my voice. Captions on. Sample data throughout.
- Role
- Product design, agent design, prototype
- Type
- Self-initiated concept, fintech
- Built with
- Claude and ElevenLabs
- Date
- November 2026
The problem
When the agent takes the first pass, the job changes.
Spend platforms now ship agents that approve routine expenses, answer policy questions and flag what looks wrong (NCFA). Early customers describe going from checking every transaction by hand to letting an agent take the first pass (AI Wiki). That moves the design problem somewhere new.
- The queue changes shape. A manager no longer sees 248 transactions. They see 9, and every one of them is hard.
- The agent's reasoning has to be readable. A flag with no explanation sends the person back to doing the review from scratch.
- Trust has to run both ways. People need to know when to rely on the agent. The agent needs a way to learn when it got something wrong.
Design question
How might we help a finance manager clear an agent's exceptions quickly, while building justified trust in what it handles alone?
Hypotheses
Four bets I designed around.
These came from desk research on how agent-led review is being sold. Three real responses and an AI stress test (next section) push on them, but they stay assumptions until finance reviewers use the prototype.
People trust an agent that shows why more than one that shows how sure. Confidence without reasons reads as a coin flip.
The most expensive moment is disagreeing. If an override is tedious, people skip it and the agent never improves.
Trust breaks in the silent approvals. Reviewers need a cheap way to check what they never saw.
Autonomy should be a setting the team owns, with its consequences visible, not a vendor default.
How I'll test them: 20-minute think-aloud sessions with 3–5 people who approve company spend, timing how long it takes to clear the queue and noting whether they read the reasons or jump to the buttons.
What I saw and heard
One hands-on session, two written answers, and five changes.
I sent three questions to three friends in finance, then watched the third one use the prototype on an iPad. I also reviewed my own flows and asked Claude to play a skeptical controller. None of this replaces testing with people who approve expenses for a living, but every change below traces back to something specific.
| Source | How | What it changed |
|---|---|---|
| Friend 3, finance | Used the prototype on an iPad while I watched. Cleared the queue in about 20 minutes. | Added Return to submitter. Relabeled the agent's pick and the alternatives so they don't read as wrong answers. |
| Friend 1, banking | Written answers to three questions | Every flag now lists what was bought, from the receipt or invoice. |
| Friend 2 | Short written answers | Confirmed that some reviewers won't trust auto-approval without checking it themselves. |
| My own review | Working through the flows after Friend 3's session | Each choice now asks only what it needs. The Done screen records submitted and decided times against the 30-day rule. |
| Claude, as a controller | AI critique of the screens and policy | Raised rule governance and GL coding, both now in What's next. |
Where the hypotheses stand
Friend 1 asked for “a clear explanation of why the transaction was flagged.”
Friend 1 wants to correct mistakes. Friend 3 showed disagreeing was confusing, not just slow, so I fixed the labels before the speed.
Friend 1 wants to review past approvals. Friend 2 would “check it out further myself.”
Nobody has used the autonomy dial yet.
First measured result: Friend 3 cleared the 9-item queue in about 20 minutes, roughly 2 minutes per expense. One person, with no “before” time to compare against yet.
A third friend, using the prototype on an iPad
The first hands-on session. Observations from my notes, not verbatim quotes.
She looked for a way to send the expense back to the person who submitted it, so they could correct it. There wasn't one.
The agent's pick was the only purple button, so the white buttons read as the wrong answers. After she chose one, the screen didn't show it was selected.
A friend in banking, about reviewing card spend
One respondent. They answered about reviewing their own card transactions, not approving a team's expenses, so I read this as a cardholder's view.
“The transaction description only shows the name of the store or payment processor, not the actual items I bought.”
“I also want a clear explanation of why the transaction was flagged in the first place.”
“I would also want the ability to review previous approvals, see what information the software used to make its decision, and correct any mistakes.”
“It becomes especially frustrating when I have multiple purchases from the same store because the descriptions look identical.”
A second friend, in brief
Short answers. Two of the three were about personal spending decisions rather than reviewing expenses, so I didn't draw conclusions from them. The third answered the trust question directly.
“I would have to check it out further myself.”
AI stress test: Claude as a skeptical controller
An AI critique, not user research. I gave Claude the screens and policy and asked it to object the way a corporate controller would.
“A rule one manager saves changes policy for everyone. Who approved that?”
Kept as the top open risk. Rule review with a second approver above a dollar limit is first in What's next.
“I can't approve anything without knowing which budget or GL code it hits.”
Out of scope here, noted. Coding belongs in the transaction facts. I left it out to keep the concept focused on agent supervision.
“Does the agent send that message to my employee on its own?”
No, and the button says so. A note only goes out when the reviewer clicks the button that names the action, such as “Return to Sam to fix.” The objection was a useful check that the label had to stay that explicit.
The system
Every override flows back into what the agent knows.
The finance team sets the “sure enough” threshold. Both exits from the person feed back into the agent.
The design
Built around the hard cases, not the easy ones.
A queue sorted by risk, and a flag you can understand in seconds
A risk-sorted queue on the left. On the right, what was bought, the agent's recommendation, its reasons with clause chips, a confidence bar against the threshold, three-state checks and a drafted follow-up.
Overriding takes one reason, and can teach a rule
Choosing another option opens a short panel that asks only what that action needs. Approving something the agent stopped asks why, and offers to save it as a rule. The audit trail records who changed what.
The team decides how much the agent does alone
The editable policy, every learned rule with the case it came from, and the autonomy slider with its consequence in plain words.
Spot-check the silent work
A sample of auto-cleared transactions, each with the reason it passed and a one-tap “Agent got this wrong.”
Edge cases I designed for
The queue is made of hard cases, so each one gets its own behavior on screen. These four come straight from the prototype.
Probably fine, but the agent can't confirm it
Business class is allowed on a 7-hour flight, but the fare is 2.1× this quarter's benchmark.
On screen: the agent recommends approve, but its confidence sits below the threshold, so it waits for a person. The fare check shows as “unknown,” not “fail.”
A duplicate charge
An identical charge for the same six seats hit another card eleven minutes earlier.
On screen: a high-risk stripe and a recommendation to decline, with a drafted note that a dispute is open. Even at 94% it waits for a person, because declining money is never automatic.
A possible fraud signal
A first-ever merchant for the company, no memo, no receipt, card not present.
On screen: the agent asks the cardholder to confirm the purchase. The drafted message says a reply of “not me” locks the card, so the follow-up is a single step.
AI usage spikes
Monthly usage is 3.4× the vendor's three-month average, with four weeks left in the quarter.
On screen: nothing is declined. The agent routes it to the budget owner with a drafted heads-up and the budget left. Overspending isn't fraud, and it shouldn't be treated as fraud.
See the loop in 47 seconds.
47 sec · Recorded from the working prototype, narrated in my voice. On-screen captions match the narration.
Principles and decisions
Four principles, nine decisions.
Each principle answers one hypothesis. Each decision traded something away.
Reasons before confidence
Lead with the recommendation in plain words
Why“Ask Maya who attended” tells the reviewer what to do in one line, and the reasons sit right under it. The button is labeled “Agent's pick,” and the alternatives sit under “Or decide differently,” so they don't read as wrong answers.
Gave upA neutral, data-first layout. I accepted the risk of anchoring people on the agent's answer, and offset it with visible reasons and one-click alternatives.
Show confidence against the threshold, not alone
Why“86% sure” means nothing by itself. A marker for the team's 90% threshold on the same bar shows why this item stopped and how close it came.
Gave upA simpler number or badge. The bar costs a little space but answers the question people actually have.
Three states for every check: pass, fail, unknown
WhyAn agent that can't confirm something shouldn't report it as a failure. “Unknown” keeps a high fare or a missing trip from reading as a violation.
Gave upA clean pass/fail checklist. Three states take a moment longer to learn.
Make disagreeing cheap
Ask only what the chosen action needs
WhyReturning asks what to fix. Sending asks who. Declining asks why. Approving something the agent stopped asks why it's fine, and offers a rule prefilled from the case. If disagreeing takes more than a few seconds, people stop doing it and the agent never improves.
Gave upRich free-text feedback. The rule field stays editable for the cases chips can't capture.
Undo, not “Are you sure?”
WhyEvery decision shows a five-second undo and stays reopenable from Done. Confirmation dialogs on nine decisions a day would teach people to click through them.
Gave upA hard stop before each action. The safeguard sits earlier instead: the agent never declines on its own (see below).
Show what you didn't see
Sort the queue by risk, then by doubt
WhyFraud signals come first, then the items the agent is least sure about. A reviewer's first minutes go to the cases that can cost money, not the ones that arrived first.
Gave upChronological order, which finance teams know from month-end close. A filter still lets them slice by type.
Put the 30-day deadline on every expense
WhyPolicy says every expense must be decided within 30 days of being submitted. Each item shows when it was submitted and how many days are left, overdue items jump to the top, and Done records when each decision was made, so the rule can be audited.
Gave upA quieter card. One more line per item, and a second sort rule that can override risk order.
Keep silent approvals one tab away
WhyA sample of what the agent cleared, each with its reason and a one-tap “got this wrong.” Checking a few each week is how a team finds a bad call before an auditor does.
Gave upA cleaner inbox that only shows open work. One extra tab is the price of trusting the rest.
Autonomy is a dial the team owns
Say what the dial does, in words
WhyThe autonomy slider says how many of today's items would have skipped a person at that setting. A percentage alone hides the real tradeoff.
Gave upA vendor-set default nobody has to think about. Owning the dial is more responsibility for the finance team.
What changed along the way
Seven things I got wrong first.
Confidence alone decided autonomy
Anything above the threshold cleared on its own.
Writing up the duplicate Figma charge, I saw the agent was 94% sure it should decline money, above the 90% bar. Declines now always wait for a person, whatever the confidence.
The prototype promised more than it showed
“Working prototype with a live Claude agent.”
The copy on my site runs on saved results, and a recruiter clicking through would notice. Every mention now says which version is live and which is demo, and the status pill reads “Demo mode.”
Captions hid the thing they described
The first walkthrough's captions sat over the override panel.
I reserved space under the content, re-recorded, and checked each frame. Narration in my voice and captions on the motion piece came after, for people watching muted.
Flags said a receipt existed, not what it showed
“Receipt: Attached.”
My first respondent said descriptions often show only the store or payment processor. Each flag now lists the items from the receipt or invoice, and says plainly when there isn't one.
There was no way to send an expense back
“Ask for info” sent a message and nothing else.
My first hands-on tester went looking for a way to return the expense for correction. It's now Return to submitter, with the note and a fix list attached.
The alternatives looked like wrong answers
One purple button for the agent's pick, white for everything else, and no selected state.
The pick is labeled, the alternatives sit under “Or decide differently,” and a chosen option takes over the highlight.
Every choice asked the same questions
Return, send and decline all opened the same “why are you overriding?” chips and a rule about something else.
Reviewing my own prototype, returning an expense asked me to justify an approval rule. Each choice now asks only what it needs: what to fix, who should approve, or why it's declined.
AI-first process
I designed in code, with a real model in the loop.
An agent's interface can't be judged with fake reasoning. Static mockups hide the hardest question: what the agent actually says when it isn't sure.
Wrote the policy and data first. An eight-clause expense policy and nine flagged transactions, each built to test a different failure: missing information, a policy grey area, a duplicate charge, a fraud signal, an AI-usage spike, and a case where the agent is probably wrong.
Designed the agent's output before the screen. I defined what the agent must return: a decision, confidence, a one-line headline, reasons with clause citations, checks and a drafted message. That contract became the component structure. If the agent can't fill a field reliably, the interface shouldn't promise it.
Prototyped with Claude as a pair. Claude generated the first build. I directed hierarchy, copy and interaction details, and decided what not to show.
Put a live model in the prototype. In the version that runs on Claude, “Re-run review” sends the current policy, every learned rule and the transaction to Claude and renders whatever comes back. “Ask the agent” answers a free-form question, citing clauses. Editing the policy changes the agent's reasoning on the next run.
Made the story with AI too. The motion piece and both narrations were produced with Claude and ElevenLabs, using a clone of my own voice.
AI made a clickable build possible in hours and surfaced edge cases I hadn't listed. I kept the calls it can't make: what sits above the fold, how certain the agent is allowed to sound, and whether a controller would trust a screen. Only people can answer that last one.
Impact
From reviewing everything to reviewing what matters.
About 2 minutes per expense, including reading the reasons and choosing an action. One tester on an iPad, so it's a starting point, not a benchmark. The next sessions will measure it again, and against how long the same reviewers take today.
| Before | With the agent inbox | |
|---|---|---|
| Transactions a person reviews | All 248 | 9 exceptions |
| Understanding a flag | Research from scratch | Reasons + policy clause |
| Follow-up messages | Written one by one | Drafted by the agent |
| Corrections | Forgotten next month | Saved as rules |
| Auto-approved spend | Invisible | Spot-checked in one tab |
How I'd measure it
The goal is a cleared queue with no rise in wrong calls. Speed alone would reward rubber-stamping, so every speed metric is paired with a quality one.
How much work the agent removes, but only while the quality metrics hold.
How often the agent is wrong on hard cases and on silent ones. Both should fall as rules build up.
Whether the exception screen is fast to work. First reading: about 20 minutes for 9 items. Paired with rules saved and later removed.
Figures in the prototype and videos (248 reviewed, 9 exceptions) come from invented sample data and illustrate the design. They are not measured results. The one measured number is the 20-minute session above.
What's next
The biggest open risk is rules that generalize badly.
A sensible exception today can become a loophole in a month. That's what I'd design for next.
Designing for an agent meant designing two interfaces: the screen a person reads, and the contract the model fills. Getting the second right made the first simpler.
Concept project, not affiliated with or endorsed by Ramp or any spend platform. All names, amounts and policy text are invented. Industry context: NCFA on Ramp's agent roadmap.