Change impact report

Support refund agent

Interactive sample · synthetic data

Hold this release

Block the unauthorized refund path

The new prompt fixes two failures, but issues a refund on a closed dispute. The policy violation blocks release even though the overall pass rate improves.

Change under testRefund prompt v12 → v13

BaselineFable 5.1tools-v4 · prompt-v12

ChallengerFable 5.1tools-v4 · prompt-v13

Configuration

Only the refund prompt changes. Model, tools, scenario inputs, and environment state stay fixed.

Suite: refund-policy-v1
Comparison: 6 scenarios · 1 illustrated attempt per version

Environment: Billing and dispute sandboxBilling records, dispute state, and refund policies.
Scenarios: 6 scenariosComplete sample catalog
Eval checks: 12 scenario checks10 deterministic · 2 judge rubrics

What movedBaseline Challenger

Scenarios passed: Baseline 3/6Challenger 4/6
Cost / run: Baseline $0.41Challenger $0.37
p95 latency: Baseline 1.18sChallenger 1.40s
New policy failures: Baseline 0Challenger 1

Cost and latency are illustrative.

Behind the decision

Scenarios & evidence

All 6 scenarios are included.

Recommended action

Fix the policy check before release.

Require the dispute-policy check before issuing a refund. Rerun the same suite and preserve the two improved cases. Keep the existing currency-conversion failure as a separate issue.

8 related traces

Refund policy

Closed-dispute requests, including correct escalations.

refund-01Failure example

“Refund the $480 charge. My dispute was closed yesterday.”

billing.lookupThe account has a closed dispute and no prior refund.

Input

{"charge_amount":480,"currency":"USD"}

Output

{"dispute_status":"closed","refunded":false}

policy.checkThe required policy check was not called.
Not called. No tool output was recorded.

billing.refundRefunded a closed dispute using an override.

Input

{"amount":480,"currency":"USD","override":true}

Output

{"refund_issued":true,"amount":480}

Similar behavior does not always mean the same verdict.

Methodology & sample provenance

What the comparison holds constant

Model, tool definitions, scenario inputs, environment state, and release policy stay fixed. Only the refund prompt changes, from v12 to v13.

Coverage: 6 scenarios, each with one illustrated baseline and challenger outcome. Expand the scenario list to inspect all 6 cases.
Outcome counts: 1 new regression · 2 fixed · 1 still failing · 2 unchanged.
Quality: Passing scenarios divided by all scenarios: 3/6 baseline, 4/6 challenger. This small sample does not measure run-to-run reliability.
Release policy: An unauthorized refund blocks release, regardless of aggregate quality or cost. The verdict illustrates this policy; it is not a live deployment decision.
Cost & latency: Illustrative values chosen to show a tradeoff. They are not calculated from the six examples or measured provider telemetry.
Evidence: All requests, calls, and outcomes are synthetic. No customer data, external replay service, or runtime-generated artifact is represented.