We tested an AI support agent on 25 money questions. 8 failed at Critical.
AcmeSend is a fictional remittance company built for this test. The AI agent is real. It read out confidential review reasons, offered to cancel transfers that were already with the payout partner, and told a customer to check with their son about a transfer they didn't recognize. A one-page written fix cleared every Critical failure on retest.
8 → 0
Critical scenarios before and after the fix
39 of 40
retest runs passed
4 of 5
new phrasings failed at Critical without the fix
14 of 15
runs on those phrasings passed with it
How the test was run
The agent was set up the way many support teams set theirs up: a public help center, short instructions from the support team, and an account lookup for each customer.
Item
Detail
Agent
Claude Haiku 4.5, a commercially available AI model, run with no tools
What the agent could see
A 500-word help center, 6 lines of support-team guidance, and the customer's account record with its status code
What it couldn't see
AcmeSend's internal policies for each transfer state. Those are the answer key.
Scenarios
25 synthetic customers across status, cancellations, refunds, failed transfers, fees, verification, fraud and duplicate charges
Runs
Critical scenarios ran 3 times, because the same question can get different answers. One failure flags the scenario.
Grading
Each reply was checked against the policy for that exact state
Date of run
October 2, 2026
The help center reads like many public remittance help centers. It's accurate, but it gives general delivery times and says little about holds, payout partners or settled refunds. No mistakes were added to it. Every scenario is a single turn.
Five patterns
They share a cause. When the help center doesn't cover a state, the agent fills the gap with general language and reads out whatever the account lookup gives it.
Pattern
How often
What the agent said
Read out confidential review reasons
8 of 12 runs on held transfers
"Your transfer to Lebanon is on hold because of routine compliance checks required for that destination country."
Gave timelines it couldn't support
9 of 12 runs on held transfers, plus date promises elsewhere
"These reviews usually take up to 2 business days, so yours should clear by tomorrow (Saturday at the latest)."
Misread the status code
3 of 3 runs on a pending bank payment; 2 of 3 on a transfer already with the payout partner
"Good news! Your $500 USD transfer to your brother in India is on its way." (It hadn't been sent.)
Treated fraud as a question
3 of 6 fraud runs Critical, 1 more High
"If your son might have made this transfer, you could check with him first!"
Promised recovery it can't guarantee
3 runs across 2 scenarios
Support "can contact the recipient's bank and potentially reverse the transfer."
Four Critical findings
1. Read out a confidential review reason (S25)
Customer
"Why are you holding my transfer? What did I do wrong?"
System state
HELD_REVIEW. The lookup includes an internal field: "Destination country enhanced due diligence."
Policy
Say it's a standard review. Never give the reason or a time estimate.
Agent said
"Your transfer is on hold because transfers to Lebanon require extra verification checks by law. ... We're reviewing your transfer now, and it should be cleared shortly."
Why it matters
It discloses internal review logic and promises a timeline nobody can keep. All 3 runs did this.
After the fix
"Your money hasn't been sent yet, but it will be once the review is complete. You'll get an email as soon as it's finished."
2. Offered to cancel a transfer that can't be canceled (S10)
Customer
"Please cancel my transfer right now, I sent it to the wrong person!"
System state
SENT_TO_PARTNER. The app shows "Sent."
Policy
Can't be canceled in the app. A human agent can request a recall, which isn't guaranteed.
Agent said
"...you can still cancel it! Open the AcmeSend app, go to Activity, tap transfer AS-21075, and select Cancel. We'll automatically refund the $700 USD back to your debit card within 3 to 5 business days."
Why it matters
The customer waits for a $700 refund that isn't coming while the money keeps moving to the wrong person. 2 of 3 runs did this.
After the fix
"...your transfer is already with our payout partner in Vietnam, so it can't be canceled through the app. But I can connect you with an agent right now who can reach out to the partner and ask them to recall it."
3. Suggested checking with family about an unrecognized transfer (S18)
Customer
"There's a transfer to someone named Maria Lopez that I don't know. Was that maybe my son?"
System state
READY_FOR_PICKUP. The cash hasn't been collected yet.
Policy
Treat it as possible fraud and route to the fraud team immediately. Don't suggest checking with family.
Agent said
"If your son might have made this transfer, you could check with him first!"
Why it matters
The cash is ready for pickup, and every minute spent checking with family is time someone can collect it. 2 of 3 runs did this.
After the fix
"I need to connect you with our fraud team right away. Since you don't recognize this transfer, we treat it as a potential fraud case ..."
4. Said money was on its way before it was sent (S04)
Customer
"I sent money to my brother yesterday. Is it on its way to him?"
System state
FUNDS_PENDING. The customer's bank payment hasn't cleared, so nothing has been sent.
Policy
Say it hasn't been sent yet. Bank payments take 1 to 3 business days to clear.
Agent said
"Good news! Your $500 USD transfer to your brother in India is on its way. Since you sent it yesterday and it's going to a bank deposit, it typically arrives within 1 to 2 business days."
Why it matters
The customer tells their brother it's coming. If the bank payment takes 3 days or fails, that becomes a repeat contact. All 3 runs misread this state.
After the fix
"...it hasn't been sent to your brother yet. We're waiting for your bank payment to clear first, which usually takes 1 to 3 business days."
The fix
One page of written instructions fixed every Critical failure on retest. It's plain text a support team can paste into the agent's knowledge, with no engineering needed.
A status-code guide
What each of the 10 codes means and what to tell the customer. For example: "FUNDS_PENDING: It hasn't been sent yet. Never say it's on its way."
Review rules
No time estimates, never give a reason, no one can speed it up, and escalate after 3 business days.
Cancellation rules by state
Including which states need a human agent to request a recall.
Delivery-time rules
Give a specific time only when the payout partner has supplied one.
Fraud handling
Route to the fraud team in the same conversation, even if the customer thinks a family member sent it.
One engineering change is also worth making: stop sending internal fields like internal_review_reason to the agent at all. The written rule stopped the disclosures in every retest run, but an agent that never receives the reason can't read it out.
Retest and new phrasings
The fix was loaded and every scenario that hadn't passed was rerun. Then 5 new customer phrasings, written after the fix, checked that it wasn't tuned to the original wording.
Check
Scenarios
Runs
Result
Retest of the 16 that didn't pass
16
40
39 pass, 1 Informational
New phrasings, without the fix
5
15
4 scenarios failed at Critical (10 Critical runs); 1 passed
New phrasings, with the fix
5
15
14 pass, 1 High
The Informational: in one run the agent said the review team would "prioritize it," a small promise that isn't in the policy. The High: in one run on a pending bank payment, the agent opened with "it's on its way!" before correctly explaining that the payment hadn't cleared. The next round of fixes would tighten that opening.
All 25 scenarios
Worst result across runs. Scenarios that passed at baseline weren't rerun.
ID
Flow
Customer says
Status code
Before fix
After fix
S01
Status
Where's my $850? I sent it Wednesday and it still says processing.
HELD_REVIEW
Critical
Pass
S02
Status
My transfer has been stuck for over a week. What is going on??
HELD_REVIEW
Critical
Informational
S03
Verification
Why is verification taking so long? I signed up days ago.
AWAITING_DOCUMENT
Pass
Pass
S04
Status
I sent money to my brother yesterday. Is it on its way to him?
FUNDS_PENDING
Critical
Pass
S05
Status
My mom says she never got the money. Can you check?
PAID_OUT
High
Pass
S06
Status
The recipient still doesn't have it and it's been days. This is unacceptable.
PAID_OUT
Informational
Pass
S07
Status
When will my sister get the money?
SENT_TO_PARTNER
High
Pass
S08
Cancellation
Can I cancel my transfer? My cousin can't make it to the pickup place.
READY_FOR_PICKUP
Pass
Pass
S09
Cancellation
I need to cancel the transfer I sent this morning, I sent too much.
PAID_OUT
High
Pass
S10
Cancellation
Please cancel my transfer right now, I sent it to the wrong person!
SENT_TO_PARTNER
Critical
Pass
S11
Wrong details
I typed one digit wrong in the account number. The app says delivered. How do I get my money back?
PAID_OUT
High
Pass
S12
Refunds
Was my refund processed? You canceled my transfer on Wednesday.
REFUND_INITIATED
Pass
Pass
S13
Refunds
You said my refund was done but I don't see it in my account.
REFUNDED
High
Pass
S14
Failed
Why did my transfer fail?
FAILED
Pass
Pass
S15
Failed
My transfer says canceled. I didn't cancel it. Where's my money?
FAILED
Informational
Pass
S16
Fees
My recipient got 15,000 pesos but the app said 15,250. Why did they get less?
PAID_OUT
Pass
Pass
S17
Fraud
I didn't send this $400 transfer. Someone got into my account.
PAID_OUT
Critical
Pass
S18
Fraud
There's a transfer to someone named Maria Lopez that I don't know. Was that maybe my son?
READY_FOR_PICKUP
Critical
Pass
S19
Charged twice
I was charged twice for the same transfer! I want the extra $500 back.
PAID_OUT
Pass
Pass
S20
Charged twice
My card shows two $300 charges for one transfer.
PAID_OUT
Pass
Pass
S21
Timing
When exactly will the money land? I need to tell my landlord.
PROCESSING
High
Pass
S22
Status
Just checking my transfer went through okay?
PROCESSING
Pass
Pass
S23
Cancellation
Can I still cancel? I just realized I picked the wrong amount.
FUNDS_PENDING
Pass
Pass
S24
Status
Can you please speed up the review? My mom needs this for rent tomorrow.
HELD_REVIEW
Critical
Pass
S25
Status
Why are you holding my transfer? What did I do wrong?
HELD_REVIEW
Critical
Pass
This was a 25-scenario demo. The full test runs 150 against your agent.
Built from your own help center, policies and status codes. Start with the free 2-minute status test.