Incident 4172: checkout p99 latency spike

Status: mitigated at 14:32, root cause confirmed. Impact: 38 minutes of elevated p99 on /checkout, 2.1% of requests timing out.

Timeline

Error rate by minute

Latency before vs after rollback

Broken spec (demonstrates inline error handling)

Root cause

svc-payments@1.88.0 introduced a synchronous fraud-check call on the hot path. See commit a1b2c3d. Follow-up: make the check async with a 200ms budget.