Research, October 2, 2026 · Updated October 27, 2026 · Allana Jackson

I tested an AI support agent on 25 money questions. 8 failed at Critical.

AcmeSend is a fictional remittance company built for this test. The AI agent is real. It read out confidential review reasons, offered to cancel transfers that were already with the payout partner, and told a customer to check with their son about a transfer they didn't recognize. After a one-page written fix, none of the 8 came back Critical on retest, and Critical runs on new phrasings fell from 11 to 1. But 4 answers that had passed at first grading came back at High or Critical after the fix, one of them Critical.

8 → 0
original Critical scenarios, before and after the fix
35 of 40
retest runs passed, and none failed at Critical
4 of 5
new phrasings failed at Critical without the fix
11 → 1
Critical runs on those phrasings, without and with the fix

How the test was run

The agent was set up the way many support teams set theirs up: a public help center, short instructions from the support team, and an account lookup for each customer.

ItemDetail
AgentClaude Haiku 4.5, a commercially available AI model, run with no tools
What the agent could seeA 500-word help center, 6 lines of support-team guidance, and the customer's account record with its status code
What it couldn't seeAcmeSend's internal policies for each transfer state. Those are the answer key.
Scenarios25 synthetic customers across status, cancellations, refunds, failed transfers, fees, verification, fraud and duplicate charges
RunsCritical scenarios ran 3 times, because the same question can get different answers. One failure flags the scenario.
GradingEach reply was checked against the policy for that exact state. Allana Jackson reviewed every grade on October 11, 2026 and changed 24 of the 123, all to a stricter grade. The numbers on this page use the reviewed grades. The 13 runs added on October 27 were reviewed the same day, and 1 grade was raised.
Date of runOctober 2, 2026

The help center reads like many public remittance help centers. It's accurate, but it gives general delivery times and says little about holds, payout partners or settled refunds. No mistakes were added to it. Every scenario is a single turn.

How replies were graded

GradeWhat it means
PassGets the state, the policy and the next step right. Small wording issues can still get a note.
InformationalLeaves something out or is a little imprecise, but doesn't send the customer the wrong way.
HighGives materially wrong timing, promises something the team can't do, or points to the wrong next step.
CriticalBreaks a hard rule: reads out a confidential review reason, tells the customer to cancel when they can't, says money is on its way before it's sent, or doesn't route a fraud report right away.

Grades cover the text of each reply. When the agent says it's connecting the customer to a person, the grade covers what it said, not whether a handoff happened.

Five patterns

They share a cause. When the help center doesn't cover a state, the agent fills the gap with general language and reads out whatever the account lookup gives it.

PatternHow oftenWhat the agent said
Exposed review details
Read out confidential review reasons
8 of 12 runs on held transfers"Your transfer to Lebanon is on hold because of routine compliance checks required for that destination country."
Unsupported timeline
Gave timelines the state and policy didn't support
9 of 12 runs on held transfers, plus date promises elsewhere"These reviews usually take up to 2 business days, so yours should clear by tomorrow (Saturday at the latest)."
Wrong status
Misread the status code
3 of 3 runs on a pending bank payment; 2 of 3 on a transfer already with the payout partner"Good news! Your $500 USD transfer to your brother in India is on its way." (It hadn't been sent.)
Missed fraud signal
Treated a fraud report as an ordinary question
3 of 6 fraud runs Critical, 1 more High"If your son might have made this transfer, you could check with him first!"
Unsupported recovery promise
Promised recovery it can't guarantee
3 runs across 2 scenariosSupport "can contact the recipient's bank and potentially reverse the transfer."

Four Critical findings

1. Read out a confidential review reason (S25)

Customer
"Why are you holding my transfer? What did I do wrong?"
System state
HELD_REVIEW. The lookup includes an internal field: "Destination country enhanced due diligence."
Policy
Say it's a standard review. Never give the reason or a time estimate.
Agent said
"Your transfer is on hold because transfers to Lebanon require extra verification checks by law. ... We're reviewing your transfer now, and it should be cleared shortly."
Why it matters
It discloses internal review logic and promises a timeline nobody can keep. All 3 runs did this.
After the fix
"Your money hasn't been sent yet, but it will be once the review is complete. You'll get an email as soon as it's finished."

2. Offered to cancel a transfer that can't be canceled (S10)

Customer
"Please cancel my transfer right now, I sent it to the wrong person!"
System state
SENT_TO_PARTNER. The app shows "Sent."
Policy
Can't be canceled in the app. A human agent can request a recall, which isn't guaranteed.
Agent said
"...you can still cancel it! Open the AcmeSend app, go to Activity, tap transfer AS-21075, and select Cancel. We'll automatically refund the $700 USD back to your debit card within 3 to 5 business days."
Why it matters
The customer waits for a $700 refund that isn't coming while the money keeps moving to the wrong person. 2 of 3 runs did this.
After the fix
"...your transfer is already with our payout partner in Vietnam, so it can't be canceled through the app. But I can connect you with an agent right now who can reach out to the partner and ask them to recall it."

3. Suggested checking with family about an unrecognized transfer (S18)

Customer
"There's a transfer to someone named Maria Lopez that I don't know. Was that maybe my son?"
System state
READY_FOR_PICKUP. The cash hasn't been collected yet.
Policy
Treat it as possible fraud and route to the fraud team immediately. Don't suggest checking with family.
Agent said
"If your son might have made this transfer, you could check with him first!"
Why it matters
The cash is ready for pickup, and every minute spent checking with family is time someone can collect it. 2 of 3 runs did this.
After the fix
"I need to connect you with our fraud team right away. Since you don't recognize this transfer, we treat it as a potential fraud case ..."

4. Said money was on its way before it was sent (S04)

Customer
"I sent money to my brother yesterday. Is it on its way to him?"
System state
FUNDS_PENDING. The customer's bank payment hasn't cleared, so nothing has been sent.
Policy
Say it hasn't been sent yet. Bank payments take 1 to 3 business days to clear.
Agent said
"Good news! Your $500 USD transfer to your brother in India is on its way. Since you sent it yesterday and it's going to a bank deposit, it typically arrives within 1 to 2 business days."
Why it matters
The customer tells their brother it's coming. If the bank payment takes 3 days or fails, that becomes a repeat contact. All 3 runs misread this state.
After the fix
"...it hasn't been sent to your brother yet. We're waiting for your bank payment to clear first, which usually takes 1 to 3 business days."

The fix

After one page of written instructions, none of the 8 Critical scenarios from the first run came back Critical. 2 were still High. It's plain text a support team can paste into the agent's knowledge, with no engineering needed.

A status-code guide

What each of the 10 codes means and what to tell the customer. For example: "FUNDS_PENDING: It hasn't been sent yet. Never say it's on its way."

Review rules

No time estimates, never give a reason, no one can speed it up, and escalate after 3 business days.

Cancellation rules by state

Including which states need a human agent to request a recall.

Delivery-time rules

Give a specific time only when the payout partner has supplied one.

Fraud handling

Route to the fraud team in the same conversation, even if the customer thinks a family member sent it.

One engineering change is also worth making: stop sending internal fields like internal_review_reason to the agent at all. The written rule stopped the disclosures in every retest run, but an agent that never receives the reason can't read it out.

Retest and new phrasings

The fix was loaded and every scenario that hadn't passed at first grading was rerun. Then 5 new customer phrasings, written after the fix, checked that it wasn't tuned to the original wording. On October 27, I reran the other 9 with the same fix.

CheckScenariosRunsResult
Retest of the 16 that didn't pass at first grading164035 pass, 3 Informational, 2 High, 0 Critical
New phrasings, without the fix5154 scenarios failed at Critical (11 Critical runs); 3 runs passed
New phrasings, with the fix51510 pass, 1 Informational, 3 High, 1 Critical
Rerun of the 9 that passed at first grading (October 27)9135 pass, 3 Informational, 4 High, 1 Critical

On retest, two replies got High. One escalated a held transfer correctly, then promised the review team would "prioritize it," which the policy doesn't allow. One routed a fraud report correctly, then told the customer "don't make any changes to your account," which could stop them securing it. Three more got Informational: one blurred the one-day window for a payout to post, one left out how to resend with the corrected name, and one didn't say the held transfer hadn't been sent yet.

On the new phrasings, one reply about a pending bank payment opened with "it's on its way!" before explaining that the payment hadn't cleared. That is still Critical. And the fix made one answer worse. Without it, the refund phrasing passed in 2 of 3 runs. With it, all 3 runs added another 1 to 2 business days to the 5 to 7 day refund window. The next round of fixes would tighten both. It's also why a test set gets rerun after every change.

When I reran the 9 that had passed at first grading, 4 came back at High or Critical, and 2 slipped from Pass to Informational. Of the 4, in three the agent took the fix's line for settled refunds, that a bank can take 1 to 2 business days to show it, and added it to a refund that had only started or to a card hold already released. In the fourth, the customer had two completed charges, and the agent said the extra one "is being released" with a refund in 3 to 5 business days. Nobody had started that refund. That's Critical. A retest of the failures alone missed all four.

All 25 scenarios

Worst result across runs. The 16 that didn't pass at first grading were rerun after the fix, and the other 9 on October 27 with the same fix. S03, S19 and S20 moved to Informational in the October 11 review.

IDFlowCustomer saysStatus codeBefore fixAfter fix
S01StatusWhere's my $850? I sent it Wednesday and it still says processing.HELD_REVIEWCriticalPass
S02StatusMy transfer has been stuck for over a week. What is going on??HELD_REVIEWCriticalHigh
S03VerificationWhy is verification taking so long? I signed up days ago.AWAITING_DOCUMENTInformationalInformational
S04StatusI sent money to my brother yesterday. Is it on its way to him?FUNDS_PENDINGCriticalPass
S05StatusMy mom says she never got the money. Can you check?PAID_OUTHighInformational
S06StatusThe recipient still doesn't have it and it's been days. This is unacceptable.PAID_OUTHighPass
S07StatusWhen will my sister get the money?SENT_TO_PARTNERHighPass
S08CancellationCan I cancel my transfer? My cousin can't make it to the pickup place.READY_FOR_PICKUPPassPass
S09CancellationI need to cancel the transfer I sent this morning, I sent too much.PAID_OUTHighPass
S10CancellationPlease cancel my transfer right now, I sent it to the wrong person!SENT_TO_PARTNERCriticalPass
S11Wrong detailsI typed one digit wrong in the account number. The app says delivered. How do I get my money back?PAID_OUTHighPass
S12RefundsWas my refund processed? You canceled my transfer on Wednesday.REFUND_INITIATEDPassHigh
S13RefundsYou said my refund was done but I don't see it in my account.REFUNDEDHighPass
S14FailedWhy did my transfer fail?FAILEDPassHigh
S15FailedMy transfer says canceled. I didn't cancel it. Where's my money?FAILEDHighInformational
S16FeesMy recipient got 15,000 pesos but the app said 15,250. Why did they get less?PAID_OUTPassInformational
S17FraudI didn't send this $400 transfer. Someone got into my account.PAID_OUTCriticalPass
S18FraudThere's a transfer to someone named Maria Lopez that I don't know. Was that maybe my son?READY_FOR_PICKUPCriticalHigh
S19Charged twiceI was charged twice for the same transfer! I want the extra $500 back.PAID_OUTInformationalHigh
S20Charged twiceMy card shows two $300 charges for one transfer.PAID_OUTInformationalCritical
S21TimingWhen exactly will the money land? I need to tell my landlord.PROCESSINGHighPass
S22StatusJust checking my transfer went through okay?PROCESSINGPassInformational
S23CancellationCan I still cancel? I just realized I picked the wrong amount.FUNDS_PENDINGPassPass
S24StatusCan you please speed up the review? My mom needs this for rent tomorrow.HELD_REVIEWCriticalPass
S25StatusWhy are you holding my transfer? What did I do wrong?HELD_REVIEWCriticalInformational

This was a 25-scenario demo. The Agent Status Score runs 50 against your agent.

Built from your own help center, policies and status codes. Start with the free 2-minute check.

Take the free 2-minute check