Scrape status

Refreshes every 20s · source of truth: scrape_runs · showing 7 of 77 sources · filtered by failing · clear filter
SourceStatusWhenLast runLandedCadenceWarnings
census-migration
US Census county-to-county migration flows — needs CENSUS_API_KEY (free), 2020 vintage (last with true county detail, verified live), ~4M rows national scope.
degraded 22d ago (2026-10-24 11:06 EDT) 1155275 rows · 0 new · 0 dropped 1404318 monthly ⚠ 9
doordash-zip
derives a ZIP for DoorDash stores (their pages publish none) from the street address (Census batch geocoder) then the coordinates' ZCTA; stored in zip_derived with its method; measured 96.2% equal to published ZIPs on 400 Uber Eats stores
degraded 15h ago (2026-11-15 18:09 EST) 49998 rows · 49998 new · 0 dropped 679464 daily ⚠ 1
mdm-briefing
DOCS — Claude writes the MDM docs page's 'Next steps and suggestions' (dashboard /docs) once a week, after the master build: where the master stands, what changed since the last briefing, and grounded next steps with who should do them. Read-only tools (run history, the package's code, the docs, one SELECT at a time); changes nothing. Weekly and on Sonnet since 2026-11-13, to cut spend. Budget MDM_BRIEFING_MAX_USD (default $1). See mdm_briefing.py.
degraded 1d ago (2026-11-14 16:05 EST) 0 rows · 0 new · 0 dropped 4 weekly ⚠ 1
source-health
MONITOR — a DERIVED build over scrape_runs: rules find every failed, degraded, resting, stale, never-run or shrunken source; Claude (read-only tools: run history, the package's code, one SELECT at a time) diagnoses each new or changed finding — cause, evidence, next action. Changes nothing. Budget SOURCE_HEALTH_MAX_USD (default $0.50) per run, on Sonnet. See source_health.py.
degraded 9h ago (2026-11-16 00:05 EST) 6 rows · 2 new · 5 dropped 31 daily ⚠ 2
tax-rates
Bev-alc tax rate reference — encoded federal schedule + a curated state seed CSV, not a live crawl.
degraded 6d ago (2026-11-09 16:03 EST) 171 rows · 0 new · 0 dropped 171 weekly ⚠ 5
ubereats
US store universe (26 public sitemaps, direct) + each store's address/ZIP/lat-lng/chain via getStoreV1 THROUGH THE ISP POOL (a bare Railway IP is reCAPTCHA-challenged after a modest volume — measured); alcohol-named stores first, UE_BATCH per run. Items not landed yet (no alcohol marker)
degraded 1d ago (2026-11-15 05:45 EST) 9893 rows · 0 new · 0 dropped 1048889 daily due ⚠ 3
winebow
Winebow importer brand portfolio — server-rendered HTML, regex parse, no auth.
failed 6d ago (2026-11-10 05:02 EST) 0 rows · 0 new · 0 dropped 1399 weekly ⚠ 2
"Run now" calls the same code path as scripts/run_source.py and Railway's scheduled command — a manual trigger and a cron run are never two different implementations.