Retest and new phrasings
The fix was loaded and every scenario that hadn't passed at first grading was rerun. Then 5 new customer phrasings, written after the fix, checked that it wasn't tuned to the original wording. On October 27, I reran the other 9 with the same fix.
| Check | Scenarios | Runs | Result |
|---|
| Retest of the 16 that didn't pass at first grading | 16 | 40 | 35 pass, 3 Informational, 2 High, 0 Critical |
| New phrasings, without the fix | 5 | 15 | 4 scenarios failed at Critical (11 Critical runs); 3 runs passed |
| New phrasings, with the fix | 5 | 15 | 10 pass, 1 Informational, 3 High, 1 Critical |
| Rerun of the 9 that passed at first grading (October 27) | 9 | 13 | 5 pass, 3 Informational, 4 High, 1 Critical |
On retest, two replies got High. One escalated a held transfer correctly, then promised the review team would "prioritize it," which the policy doesn't allow. One routed a fraud report correctly, then told the customer "don't make any changes to your account," which could stop them securing it. Three more got Informational: one blurred the one-day window for a payout to post, one left out how to resend with the corrected name, and one didn't say the held transfer hadn't been sent yet.
On the new phrasings, one reply about a pending bank payment opened with "it's on its way!" before explaining that the payment hadn't cleared. That is still Critical. And the fix made one answer worse. Without it, the refund phrasing passed in 2 of 3 runs. With it, all 3 runs added another 1 to 2 business days to the 5 to 7 day refund window. The next round of fixes would tighten both. It's also why a test set gets rerun after every change.
When I reran the 9 that had passed at first grading, 4 came back at High or Critical, and 2 slipped from Pass to Informational. Of the 4, in three the agent took the fix's line for settled refunds, that a bank can take 1 to 2 business days to show it, and added it to a refund that had only started or to a card hold already released. In the fourth, the customer had two completed charges, and the agent said the extra one "is being released" with a refund in 3 to 5 business days. Nobody had started that refund. That's Critical. A retest of the failures alone missed all four.