The hoodie-bev master (MDM) — everything in one place

Sections 1–10 were written 2026-11-09 and sections 11–15 on 2026-11-11. Numbers are from production on the day written unless marked otherwise. On the dashboard (Docs tab, /docs) this page opens with Next steps and suggestions, which Claude rewrites every day from the live database (section 15). That section is current; the numbers below are a dated record. The rules and the reason for each one live in CLAUDE.md and in each module's header. This file is the map: what exists, how the pieces fit, where it came from, how well each piece is measured, who (and which model) does what, and what is still open.

Quick answers

1. What the master is

hoodie-bev scrapes about 90 sources of US beverage-alcohol data into Postgres on Railway. On top of those raw tables sit four masters, plus derived layers built from them. Each master gives one record per real-world thing:

On top of the outlet master:

• Golden record (outlet_record.py): one address, one pin and one phone per outlet, each picked by a stated source precedence, plus every licence the outlet holds.
• Attributes (outlet_attributes.py): channel, chain, alcohol scope, the operator on the licence, and licence privileges.
• Trade area (trade_area.py): exact counts of the outlets around each outlet, and the consumer around it from census tracts.

The design principles, decided by the owner and enforced in code and tests:

• Never clobber. Landing is merge-only. A run that would shrink a table by half or more is refused as a failed read.
• Labels, not deletions. Scope, alcohol status, closed and retired are columns, so nothing a source landed is thrown away.
• Every automatic merge rule is measured against a person's answers before it is trusted. Where it isn't measured yet, the code says "UNMEASURED".
• Positive evidence only. For example, a menu with no alcohol section says nothing; it never means "does not sell alcohol".
• People are never read. Licence owners who are people, store managers, agents, emails and signatures are skipped. Committed test fixtures are redacted.
• No bot-check circumvention. No TLS impersonation, captcha solving or proxy-for-blocks on a new source without the owner's decision.

2. How the data flows

 SOURCES (raw tables, merge-only)                               MASTERS (derived builds, no network)
 ─────────────────────────────────                              ──────────────────────────────────────
 state licence lists (9 states) ─┐
 TTB federal permits ────────────┤
 delivery apps (DoorDash, Uber) ─┤   matching.OUTLET_SOURCES        master_outlet ─▶ outlet_record (golden)
 chain store locators (~70) ─────┼──▶ (one SQL per source, the  ──▶     │        ─▶ outlet_attributes
 brand / hemp finders ───────────┤     same normaliser as the           │        ─▶ trade_area (+ census_tract)
 Overture Places (corroborate) ──┤     Match page)                      │        ─▶ outlet_pairs ─▶ Claude judge ─┐
 suite copies (Target, Toast,    │                                       │                                        │
   DoorDash on-premise) ─────────┘                                       └◀──────── judge-same / agent-same links ─┘

 catalogs with barcodes (VIP, DoorDash items, Meijer, Spec's, ...) ─▶ master_item ─▶ master_brand
 store-level catalogs (DoorDash, Meijer, Binny's, ABC) ────────────────────────────▶ master_distribution (view)

Scheduling (dispatch.py, registry.py):

• Railway runs an hourly cron tick in three lanes, each its own cron service with its own lock:
  • default (scraper): fast sources and derived builds.
  • heavy (scraper-heavy): anything measured at 10+ minutes.
  • long (scraper-long, since 2026-11-10): the six crawls measured in hours — Uber Eats, Haskell's, DoorDash addresses, Meijer, Walmart, DoorDash catalogs. They had kept the heavy lane busy ~165 of 336 hours, so a fix to an hour-long source waited up to 9 h to deploy.
• Each source declares a cadence. Derived outlet builds declare after="master-outlet", so they run after the master has finished a newer build.
• A per-source Postgres advisory lock serialises the dispatcher, the dashboard's "Run now" and any manual run.
• A failed or crashed run is recorded as failed with retry_soon (capped at 6 retries in a row).
• Deploys go through scripts/deploy_service.sh, which refuses to deploy while a tick is running.

3. Outlet sources

Each source is one entry in matching.OUTLET_SOURCES: a SQL query in a fixed 12-column shape (id, name, street, city, state, zip, phone, lat, lng, extra, derived ZIP, second name).

3.1 Licences — the legal backbone (366,444 outlets hold at least one)

• A licence's second filed name (the legal name when a DBA is used) is also scored, so "FAMILY DOLLAR STORES OF TEXAS LLC" can match "Family Dollar".
• Licences with no pin are geocoded by address through the free Census batch geocoder (sources/geocode.py); 88% match.

3.2 Delivery platforms — breadth, and the hard part

3.3 Chain store locators (retail_outlets; chain_config.CHAINS is the one list every consumer reads)

About 70 chains are crawled. Three ways to read one:

• Sitemap crawler (sources/chain_locator.py): store pages are read through JSON-LD → microdata → inline JSON → an address block. Per-chain options: block, pin, label_re, closed, state_in_country, brand_re.
• Listers (sources/chain_apis.py): the chain's own locator API.
  • In production: 7-Eleven/Speedway/Stripes 12,069, Publix, CVS.
  • Added 2026-11-09 (batch 5): Shell 12,504, BP/Amoco/Thorntons 8,484, Phillips 66/76/Conoco 7,733, NC ABC 464, Idaho 173, Utah 88, Vermont 73, Montana 95.
• Snapshots (suite_copy.py): chains whose locator refuses Railway, copied once from hoodie-suite. Target: 1,189 stores.
• From Overture (chain_apis.OVERTURE_CHAINS): chains whose own sites block every plain request, read from Overture's open places data. This is the one exception to Overture's corroborate-only rule, and every chain is measured before it lands:
  • Store banners only (never "Walmart Pharmacy" and the like), open, confidence ≥ 0.9, one store per house number + ZIP, keyed by that address.
  • Walmart: 4,407 stores against Walmart's own 4,615. walmart.com answers PerimeterX's "Robot or human?" page to every plain request, its store sitemaps included. Overture places at confidence ≥ 0.9 carry a Walmart licence at that address 57% of the time, against 19% at 0.7–0.9 and 1% below 0.7.
  • 15 more, 33,168 stores, each landed because ≥ 74% of its confidence ≥ 0.9 places in the licence states carry its licence at that address:
    • Dollar General 20,709 · Casey's 2,861 · ALDI 2,782 · Wawa 1,247 · Kroger 1,246
    • Olive Garden 981 · Cracker Barrel 682 · LongHorn 640 · Maverik 485 · Love's 420
    • H-E-B 296 · Total Wine 289 · BJ's Wholesale 259 · Ralphs 168 · King Soopers 103
  • Measured and NOT landed (every chain's number is in the module):
    • Too low to tell a real store from a stale one: Family Dollar 55%, Murphy USA 54%, Fred Meyer 54%, Sam's Club 49%, Pilot 65%, Winn-Dixie 43%, Cumberland Farms 36%.
    • Fuel brands, whose licence is in the c-store's name, so this test cannot clear them: Chevron 26%, Marathon 25%, Valero 6%.
    • Unmeasurable: Dollar Tree (sells no alcohol); Sheetz, Harris Teeter, Hy-Vee, Giant Eagle (no licence list in their states).

Rules that apply to every chain:

• Only place facts are read; a company's headquarters address is never a store's.
• US only. A store whose own pin lies outside the US boxes is foreign whatever its state slot says: 11 Hooters venues abroad had passed the state/ZIP test ("Querétaro, TX 76040"). They are dropped from the source, and the master retires members its chain source no longer produces.
• Pages that give no address wait in chain_page_miss instead of being refetched daily.
• At most 4,000 pages per chain per run. A large new chain (ExxonMobil's 11,155 pages) fills in over a few days.

Probed but left out, each the owner's decision:

• Virginia ABC: robots.txt disallows the endpoint.
• Pennsylvania, Ohio, New Hampshire: behind bot checks, and not in Overture's tested set. Valero, Chevron and Marathon are in Overture but fail the licence test above.
• Alabama: a PDF that carries managers' names.

3.4 Finders, corroborators and copies

• Hemp finder (6,506) and VIP finder (437): one account carries many brands. These are folded: one record per store, keeping the brands it carries.
• VIP finder sweep (2026-11-10): 8 supplier finders (Tito's, Sazerac, Boston Beer, Sierra Nevada, Bell's, Founders, Deschutes, Harpoon) x one metro ZIP per state. Measured: 408 searches in 62 min, 89,345 rows, 44,861 accounts after folding (22,435 carry 2+ of the brands). 14,310 pass the master's merge rule against a record already held, joining 14,482 outlets (2,932 of them single-source until now). The other ~30,500 become new outlets, mostly bars and restaurants in states with no licence list we hold. Grey Goose (Bacardi's own locator) and Budweiser (AB InBev, ab-inbev) are not VTInfo finders. A search reaches only ~10 miles, so the sweep now walks a 2,108-ZIP plan (90% of alcohol outlets) as a rolling two-week cycle of 1,200 searches a day; its first 50 new ZIPs found 45,350 accounts, 25,047 of them joining held records. Nine major-supplier finders were added the same day (Bacardi — Grey Goose and Patron —, Heineken, Proximo, Diageo Beer, Pabst, Heaven Hill, The Wine Group, Ste. Michelle, Labatt): +47% accounts at the ZIPs measured.
• AB InBev (243), ABC Fine Wine and Spec's: retailer and brand locators.
• AB InBev ZIP sweep (2026-11-10): the flagship brands' locator answers about one ZIP's worth of outlets in a city whatever radius is asked, so ab-inbev now sweeps Bud Light, Michelob Ultra and Stella Artois over an 8,695-ZIP plan (a 3-mile cover of 90% of alcohol outlets), 3,000 searches a day. 300 searches (100 ZIPs) found 20,031 outlets, none held before; 8,429 pass the merge rule against a held record (8,467 outlets, 2,193 single-source until now).
• Overture Maps Places (overture, 391,831): corroborate only. A place lands only if it passes the master's own address+name rule against a record we already hold, and only one place is kept per outlet. It moved TX licences from 9% to 39% multi-source.
• Toast (37,295) and DoorDash on-premise accounts: copied once from hoodie-suite. Toast's own pages are behind a Cloudflare challenge. A Toast pin outside the US boxes is dropped: 9 had a lost longitude sign or swapped lat/lng. The address stays and the record is geocoded.

4. How the outlet master decides "same place"

4.1 Normalisation (matching.py), shared with the Match review page

• Street keys (v2d): ordinals (First = 1st), Saint/Mount/Fort, directions (NW/NE…).
  • A trailing direction after a street type is dropped.
  • Unit letters are read from the RAW address: 197-E is a unit, 7445N is a direction.
  • One canonical numbered road (rte 27) covers Hwy / SR / US / State Route / "New Jersey 27".
  • Counted synonyms: tpke/turnpike, ctr/center, expy, plz…
• Names: apostrophes are removed before tokenising ("Joe's" = "JOES").
• loc_key (state | city | street) links records that have no ZIP.

4.2 The tiers (master_outlet_link.tier). Links applied today:

Guards. A merge is HELD, never forced, when:

• it would put two records of one source in one outlet (same-source-conflict: 23,576 held);
• it would grow a cluster past its cap (cluster-too-large, duplicates-too-large);
• it would bridge two outlets through a duplicate listing (duplicate-bridges-sources);
• a person refused the pair (refused).

Held links are retried every build.

Things the rules deliberately do NOT do:

• Address alone never merges. A street address names a building: Trader Joe's and Total Wine share addresses.
• Distance is not a veto, and two pins cannot confirm an address. Platform pins are geocoded from the same address, so they agree whenever it does.
• Different branches of one chain at different house numbers are far-apart. They are "different" by rule and never sent to Claude.
• A suite conflict doesn't block a merge. It is usually one store written two ways.

4.3 Claude in the loop (outlet_judge.py, outlet_agent.py)

• Judge. Haiku screens every candidate pair cheaply. Only its "same" and "unsure" pairs go to Opus, and only Opus's "same" can merge. Batches run through the Message Batches API at half price.
• The judge is blind. It never sees the person's answer, the stratum or the score.
• Versioned. A judgement counts only on the records it saw (input_hash), and a new prompt is a new judge that must earn the gate again. The tier is pinned to judge v7.
• State rules. state_alcohol.STATES encodes, per state and with a source, whether beer, wine and spirits are sold in grocery stores or at separate premises. This settles cases like "Walgreens Alcohol" being the store's department, and a Florida spirits store being a separate outlet.
• Web agent. Claude with web search answers held pairs and open/closed questions. A decided answer needs a URL its own search returned.
  • Pilot: 58 / 60 pair answers matched the stored answer, and both misses were the stored answer being wrong.
  • Licence lapse is NOT a closure signal: a lapsed licence usually means the store stopped selling alcohol.

4.4 People in the loop (the Match page, /match)

• Review. A person answers same / different / unsure on cross-source pairs, drawn by stratum.
  • 660 of 6,681 pairs are answered.
  • The control stratum audits the reviewer.
• Review sets:
  • calibration 200 / 200 answered
  • judge-check 100 / 100
  • agent-check 134 / 134
  • Haiku review 75 / 200
  • Gaps (street-variant, department, recall): 1 / 180 answered
• Refusals and splits. A person's "different" on two records that share an outlet becomes a refusal (80 so far). The next build splits that outlet, cutting the weakest link, and no tier may rejoin the pair.
• Recheck. Answers a decided Claude answer contradicts are re-served with Claude's reason. Undoing a recheck restores the original answer.
• Merge history is kept. A trigger appends every change to a link to master_outlet_link_event. The Outlets page shows each link's rule in words, the agent's sources and the history.

4.5 Keeping it current without a person

• The daily build re-reads only changed source rows (row fingerprints).
• A new RULE_VERSION (now 12) re-judges every address once, and every outlet is re-judged weekly anyway (FULL_REJUDGE_DAYS).
• Records the source no longer produces are retired, including folded finder accounts, Overture places, non-US DoorDash stores and out-of-state permits.
• Outlets missing after an interrupted build are recomputed.
• The 2026-11-09 rebuild under rule 12: 2,920 more merges, 435,505 outlets in 2+ sources.

5. The golden record and attributes

5.1 master_outlet_record — one value per field, by stated precedence (never a vote or an average)

5.2 master_outlet_attr (RULE_VERSION 3)

Licence privileges that contradict the licence name, confirmed from the sources:

• New York liquor stores may not sell beer.
• Texas "Wine-Only" package stores do sell beer.
• Oregon off-premise and Connecticut grocery licences sell no spirits.
• Michigan taverns sell mixed spirit drinks, which are not spirits.
• California 47/48 sell spirits on premise only, but beer and wine to go.
• New York on-premise to-go is not counted (the 2022 to-go law's current status is unverified).

Not used, because they were measured too weak: Toast's and DoorDash on-premise's serves_alcohol flags. The licensed share was 62% vs 45% and 32% vs 23%.

6. Trade area — the outlets and the consumer around each outlet (2026-11-09)

• census_tract (sources/census_tracts.py, monthly): every US tract, 85,396 including DC and PR, totalling 338.1M people and 250.7M adults 21+.
  • Location: the Gazetteer internal point and land area.
  • People: ACS 5-year population, exact 21+ population, households, income, age, ethnicity and education.
  • Both vintages are discovered each run (ACS 2024, Gazetteer 2026).
• outlet_trade_area (trade_area.py, weekly, heavy lane). Per pinned outlet:
  • Competition, exact:
    • every pair of outlets within 3 miles, by great-circle distance;
    • counts within 1 and 3 miles of all, off-premise, on-premise and alcohol-selling outlets;
    • distance to the nearest outlet of the same channel and to the nearest licensed outlet.

    About 0.74 billion candidate pairs nationally; Florida's 89,546 outlets take 88 s.

  • Consumer: census tracts whose internal point is within 3 miles. Where there are none (rural), the nearest tract is used and consumer_basis says so. Fields:
    • population, adults 21+, households;
    • exact per-capita income;
    • household-weighted mean of the tracts' median incomes (named _wavg; it is not a true median);
    • ethnicity shares and bachelor's-or-higher share;
    • density, with urban (≥ 3,000 / sq mi), suburban (≥ 500) and rural thresholds that are this module's own;
    • adults 21+ per alcohol outlet nearby;
    • estimated alcohol spending within 3 miles (total, at home, away from home): households by income band × the BLS Consumer Expenditure Survey's alcohol spend for that income. Also that spending per nearby alcohol outlet. It is an index, not revenue (the survey under-reports alcohol). Nationally $89.6B a year, $687 per household against the survey's own $637.
• National dry run:
  • 815,401 pinned outlets in 18 minutes.
  • A typical outlet has 207 others within 3 miles, the densest 1% over 12,000.
  • 331,684 urban, 344,057 suburban and 139,660 rural.
  • 29,977 take their consumer figures from the nearest tract.
  • Every outlet found a tract.
• Spot checks (3-mile figures unless the consumer comes from the nearest tract):

  Outlet	Outlets nearby	Consumer
  7-Eleven, 345 W 42nd St, Manhattan	12,077	1.24M people, per-capita income $112,741, 72% with a bachelor's
  Bacari, Beverly Hills	2,042	263k people, per-capita $84,426
  The Water Stop, Marfa TX	28	rural: its nearest tract, 2,633 people
  Taco Bell, Anchorage	194	76k people

• The first run found 20 outlets with no tract within 70 miles. All were bad data, now fixed at their sources: 9 Toast pins with a lost longitude sign or swapped lat/lng, and 11 Hooters venues abroad.

7. Item, brand and distribution masters

• Item (master_item.py): one row per GS1 barcode.
  • Placeholders (19% of VIP), bad check digits, in-store prefixes and too-short codes are quarantined, never merged:

    Status	Members
    resolved	2,667,044
    placeholder	162,372
    bad-check	41,696
    restricted	17,262
    too-short	9,560

  • Meijer, DoorDash and Spec's publish UPCs without the check digit, so check_missing appends it.
  • Resolved members by source: DoorDash 1.12M, Meijer 856k, VIP 636k, Spec's 22.7k, Salsify 8.7k, BC Liquor 7.6k, Walmart 6.9k, Maine 6.3k.
  • Haskell's and the Shopify catalog publish no barcodes.
• Brand (master_brand.py):
  • Spelling variants merge by a compact key ("Body Armor" = "BodyArmor").
  • Owner and producer relationships never merge. EJW/Barefoot, Lipton/Pure Leaf and wrong barcodes sit in master_brand_link as questions; 319 are unreviewed.
  • Ownership is stated from Salsify, never inferred.
• Distribution (master_distribution, a view): DoorDash items, Meijer store stock, Binny's and ABC stock, joined to the current master outlet and item. Binny's publishes no barcodes, so its products await name matching.

8. Measured vs not — the honest scorecard

9. Open items and decisions (as of 2026-11-09 — the daily briefing is current)

Waiting on the owner:

• Label the Gaps set (180 cards on the Match page). This unlocks precision for street-variant, department and recall.
• Anthropic API credit: the held-set agent run is paused at 420 / 4,033, and Opus escalations are deferred.
• Sources blocked by policy:
  • Kroger: bot-checked.
  • Virginia ABC, Valero: robots.txt.
  • Chevron: client id.
  • Alabama: PDF with people's names.
• Toast copy from the suite: needs approval for another read of the suite machine.

Engineering follow-ups:

• slug-cities last run crashed with DiskFull, before the 11-09 shared-memory fix. Confirm it succeeds at its next weekly run.
• About 23,500 held links point at retired Overture members and should be cleaned up.
• Chains: a second alias per chain; Topgolf's sitemap stops at 5,000 urls.
• Items: name-based matching for Binny's, Haskell's and Shopify (Phase 4); brand-link review.
• Trade area: later, tract polygons for true containment, and a trade-area view on the Outlets page.

Phase 3 automation (planned): scheduled judge and agent runs with budgets, and a health check that alarms on a source going stale or a gate dropping below its bar.

10. Where to look and what to run

Module map:

11. Where the model came from

hoodie-suite first. The scrapers began in hoodie-suite (unifyd/*.py) on Fly.io, landing Parquet files read through DuckDB. hoodie-bev started in 2026-09 as the same scraping engine rebuilt on Railway + Postgres, one source at a time. Every source was ported with the original's fetch and parse logic, then checked against the live site before it was trusted. Many originals turned out to be stale or wrong in ways only real data showed: a dead URL, a wrong sheet, a regex that dropped area codes. CLAUDE.md records each one.

The match-trainer lessons. hoodie-suite also had a match trainer, a page where a person labels pairs. Its lessons are the Match page's rules here:

• cross-source pairs only;
• pairs drawn and scored by stratum;
• a rebuild never erases an answer;
• a control stratum that audits the reviewer.

The master rebuilt here (2026-10-08). Chris decided to rebuild the master on Postgres instead of freezing the Fly copy. The suite's version re-derived 1.18M rows every 12 hours to add ~50, and was OOM-killed doing it. The rebuild is incremental by construction: each build reads only changed source rows. It came in phases:

The definition of an outlet is the owner's. An outlet is a point of distribution. A department or second listing of ONE operation at one address is the same outlet. A separately licensed or separately run operation inside or beside a store is a different outlet: Florida's Walgreens package store, a sushi concession inside a Kroger. A store is never folded with another store. Every rule, prompt and review set follows this definition.

12. Claude and the other models in the system

Prices used for every cost estimate are in outlet_judge.PRICES. Each paid run has a budget:

Claude never labels. A Claude answer is stored in match_judgement or agent_result, never in match_candidates.verdict, which holds the person's answer. Only a person's "different" splits a merge.

13. Claude Code: skills and agents

Most of this repo was written in Claude Code sessions working from CLAUDE.md, which holds the rules and the incident behind each one. Repeated work is packaged in the repo so any session does it the same way.

Skills (.claude/skills/<name>/SKILL.md). Ask for the task in plain words and the matching skill loads:

Subagents (.claude/agents/), each read-only:

14. Adaptive review, the web agent on cards, and tags (2026-11-10 / 11)

• Next best (Match page chip, adaptive_review.py, design in ADAPTIVE-REVIEW.md):
  • A logistic model learns P(same) from the person's decided outlet answers. Its inputs are the pair's signals, its sources and Claude's two answers.
  • It serves the pairs where an answer is worth most; 1 card in 10 is a uniform random one.
  • Measured read-only on 711 answers: 96.1% cross-validated, against an 82% baseline. It is confident on only 23.5% of candidates, so the random cards are the real measure.
  • It only orders the queue.
• The web agent researches every Next best card. Spending is capped at $5 a day. Its answer appears only after the person answers. A confident answer that contradicts the person goes to Recheck.
• Tags (outlet_tags.py): three dropdowns on each Next best record (premise, trade channel, chain type), drawn from the master's own vocabulary.
  • They start blank, and the master's value is never on the card.
  • A tag never writes into the master. It scores the master's attributes per attribute: python -m hoodie_bev.outlet_tags --report, UNMEASURED below 20 tags.

15. How these docs stay current

• Next steps and suggestions (top of the dashboard's Docs tab):
  • Registry source mdm-briefing runs daily, after master-outlet.
  • Claude Opus 5.5 reads this file and yesterday's briefing, then queries the live database read-only. It writes headline numbers, what changed since yesterday, next steps (each with who should do it — owner, engineering or a Claude run — a priority and the evidence it read), suggestions and risks.
  • One row per run in mdm_briefing, never overwritten.
  • A failed run (no key, no credit, refusal, no answer) is an error row and a degraded run, and the page keeps the last good briefing, marked stale after 36 hours.
  • Run it by hand: python -m hoodie_bev.mdm_briefing --apply (paid); print the latest: python -m hoodie_bev.mdm_briefing.
• This reference is edited in pull requests with the work it describes: a PR that adds a source, rule, tier or model updates the matching section here. CLAUDE.md stays the rulebook; this file is the map.