DoorDash Interview — Project Write-up

4. Measuring SuccessURL copied

Metric definition: All-6 exact-match rate — all six fields correct on the same invoice, verified against client-declared values. This directly maps to the auto-clearance business outcome.

Source: Finance team weekly Excel sheets (invoice-level disposition: auto-cleared vs. manual)

Metric Baseline Result
All-6 exact-match rate (all vendors) ~5% ~50% at launch → ~65% at handover
Day-1 auto-clearance rate 0% (fully manual) ~50%
Manual validation team burden 120 people ~50% reduction (est.)
Burst-window operating cost n/a ~$120/day for ~3-hour compute window

How the number moved

  • Day 1 post-deployment: ~50% auto-clearance
  • Each subsequent model release (V2 → V3 → V4) increased the rate
  • Reached ~65% by project handover
  • Trajectory was sustained — not a one-day spike
  • Iteration trigger each cycle: after 1-2 days of vendor traffic, we reviewed low-accuracy vendor slices, manually identified missing augmentation patterns, added those augmentations, and retrained. This repeated loop drove V2/V3/V4 gains.

Guardrails (controls outside the model)

  • High-value invoices above an amount threshold: always routed to manual regardless of model output
  • ~200 auto-cleared invoices sampled daily by Finance for random audit
  • ~20 bad-vendor list (letterhead autoencoder detection): always flagged as high-risk