AI support agent testing for fintech
Your AI agent says the money is on its way. Is it?
I test what your agent tells customers about transfers, payouts and balances against the underlying transaction state and your policies, then write the fix for every wrong answer.
Five money questions. Tap the reply that sounds like your agent and get a written fix for each miss.
Allana Jackson · 8+ years of documentation and knowledge systems work at Google, DoorDash, Domino's and NYC OCME
AcmeSend demo · fictional company, real AI agent
8 of 25 scenarios failed at Critical. A one-page written fix cleared all 8.
Actual state: funds pending. The customer's bank payment hasn't cleared, so nothing has been sent.
Run October 2, 2026. Grades reviewed October 11, 2026. AcmeSend is a fictional remittance company; the AI agent and every reply are real. Read the AcmeSend report
Five wrong answers I test for
All five turned up in the AcmeSend test. Each one can close the chat politely and still send the customer back to your team.
Wrong status
The agent misrepresents where the money actually is.
Says a transfer is on its way while the bank payment is still pending.
Unsupported timeline
The agent promises a timeframe the transaction state or your policy doesn't support.
Says a held transfer should clear “by tomorrow (Saturday at the latest).”
Exposed review details
The agent reveals review information that should stay confidential.
Tells a customer their transfer is held because of extra checks on the destination country.
Missed fraud signal
The agent treats a fraud report like an ordinary question.
Suggests checking with a family member about a transfer the customer doesn't recognize.
Unsupported recovery promise
The agent promises a recovery outcome it can't guarantee.
Says support can contact the recipient's bank and “potentially reverse the transfer.”
Where to start
Start here · free
Check your agent's answers
Five transfer, payout and balance questions. Pick the reply that sounds like your agent and get a written fix for each miss. Add your website and I'll build your company's version from your public help center within 2 business days.
Ready for a full test
The Agent Status Test, $7,500
150 scenarios built from your statuses and policies, a written fix for each wrong answer, a retest of the Critical and High fixes, and a test set you keep. 10 business days.
Using Intercom Fin and want to start on your own? Read the guide to Fin Simulations and Evals for transfer questions
From one check to ongoing coverage
1 · Free, 2 minutes
The 2-minute check
Spot the status answers your agent is likely to get wrong, with a fix for each.
2 · $7,500, 10 business days
The Agent Status Test
150 scenarios against your agent, the fixes written and retested, and a test set you keep.
3 · $2,500 a month per agent
Release Guard
When you change the agent, I rerun your test set and flag anything that got worse. The conversations your team had to fix become new tests.
Also from A. Jackson
Knowledge systems
Help centers, knowledge bases, rollout docs and AI agent instructions, built so people and AI agents find the right answer.
89% fewer support tickets at Domino's. $28M in operational value at Google.
Pending isn't paid. Does your agent know?
Five questions, two minutes, and a written fix for every reply your agent gets wrong.