Source registry

One place that says what we pull from, how, what we may do with it, and what the master may let it decide — read by the CRM, the master build and hoodie-suite, instead of each keeping its own answer.

This document is the design and the first slice. The code is src/hoodie_bev/source_record.py (the shape) and src/hoodie_bev/registry_reconcile.py (the comparison). The measured state of the two existing registries is registry-reconciliation.md, generated from the snapshot in registry-snapshots/.

Decisions (made 2026-10-26)

question	decision	why
Where does it live?	hoodie-bev.	It is where the scrapers, the run ledger and the master are going (Fly is being sunset). The registry should sit next to the ledger it reports on.
How do consumers read it?	A versioned read API.	hoodie-crm and hoodie-suite are different repos and owners; the only coupling allowed between them is versioned data over HTTP, never a shared import.
What is the first slice?	Schema + reconciliation.	Read-only, no behaviour change. It shows how far apart the two registries already are before anyone commits to a design.

What is in this slice, and what is not

In: the v1 record and its validator; projection of both existing registries onto it; a report of agreement, disagreement and silence.

Not in: any change to what registry.py or the dispatcher does; populating rights, trust or shape.keys for any source. The last is deliberately not a code task (see "Open decisions"). Slice 2 (persist + serve) is described under "Persist and serve" below.

The record

One record per source or build. Four parts, because today they are tangled in the suite's one dict and mostly absent here.

part	answers	examples
identity	what is it	id, kind (source / build), label, status (live / parked / disabled / archived), replaced_by
acquisition	how do we get it	where, lane, cadence, interval_h, cost_class, requires (env-var names)
rights	what may we do with it	record_ref (the verbatim ToS + robots record), facts_use, terms_url, needs_counsel
trust	what may the master let it decide	master_role, spine_eligible
shape	what does it give us	grains, tables, keys (identifier kinds: upc, gtin, license_no, place_id, vpid …), raw_fields_ref

Acquisition is engine-private; a consumer should not need it. Rights, trust and shape are what the CRM and master read.

Derived facts are not stored

Last success, freshness against cadence, row counts, coverage, fill rates: all computed from the run ledger and the warehouse when asked, and served beside the record. A hand-kept last_ok is a lie waiting to happen.

Rules the validator enforces

• Unknown is None (or empty), never a guess. False and 0 are answers; empty is silence. The fill-rate report counts only real values.
• requires holds names, not values. Anything not shaped UPPER_SNAKE (a NAME=value, lower-case, spaces, empty) is rejected, and the message does not echo it back. This is a shape check, not a secret scanner: an upper-case token that is itself a secret would pass.
• A build may carry rights. It inherits the rights of what it reads: the suite's own model treats a hash or embedding derived from an asset as a derivative work, gated like the asset. No suite build carries a rights record today (dam-gallery, the derivation over DAM assets, is a source in the suite's registry), so this rule is permissive rather than demanded by current data. Trust for a build is undecided until someone decides it.
• Only an identity-seed can be the spine (a missing role does not qualify either). Standing rule: TTB is a tier-0 contributor and never the spine; VIP/vpid is a corroborating key only.
• replaced_by only on archived. Archive, never two live entries for one source.
• contract() refuses invalid records and duplicate ids rather than serving or de-duplicating them. validate() reports a wrongly typed list field instead of raising.

The envelope

{ "schema_version": "1", "generated_at": "2026-10-26T00:00:00Z", "sources": [ { "id": "...", "kind": "...", "acquisition": {}, "rights": {}, "trust": {}, "shape": {}, "legacy": {} } ] }

Consumers pin schema_version and refuse a version they were not written against. Adding an optional field does not bump it; removing, renaming or changing the meaning of a field or enum value does.

How the two existing registries map

Only where the two vocabularies mean the same thing:

suite	v1
id, label	id, label
enabled true / false	status live / disabled
cadence, interval_h, cost_class	same names (a value outside the enum is carried in legacy, not coerced)
requires	acquisition.requires
tables	shape.tables
rights_record	rights.record_ref
klass="build" or listed in BUILDS	kind="build"

Everything else (klass, source_class, note, caps, shards, catalog_map, session_budget …) goes to legacy, verbatim and unvalidated. In particular klass="mac" is not turned into where="residential": related, not the same claim. The report lists any shared entry where the suite says mac and this repo says where="cloud", so a placement is decided on purpose (there are none today).

hoodie-bev's Source maps one to one: every entry is status="live", where/lane/cadence carry over, table becomes shape.tables, note goes to legacy. The four DERIVED builds (master-brand, master-item, master-outlet, slug-cities) are kind="build"; a test fails if a note says DERIVED and the id is not in that set. That catches a build whose author used the word, not one described differently (hemp-scan's note says "no fetch, Postgres read + regex classify" and it is still a source until someone decides).

What the reconciliation found (2026-10-26, suite origin/main @ 65ba01da)

Regenerate rather than trust this paragraph: python -m hoodie_bev.registry_reconcile --suite docs/registry-snapshots/suite-65ba01da.json.

• 92 suite entries (77 sources + 15 builds) against 48 here; 32 share an id, 60 are suite-only, 16 are here only. No two ids differ only by case or punctuation.
• 14 disagreements on shared ids: cadence 6 (e.g. census, haskells, meijer, geocode), landed table 5 (e.g. binnys, vtinfo, ubereats), enabled-vs-live 3 (doordash-addr, geocode, ubereats are disabled in the suite and run here). Some of these are the port working as intended; each still needs a person to say which side is current.
• Neither registry says anything about rights beyond two suite entries (one record, dam-bacardi, shared by dam-bacardi and dam-gallery), and nothing at all about trust, grains or identifier kinds. rights.record_ref is set on 2 of 92 suite entries; facts_use, terms_url, needs_counsel, master_role, spine_eligible, grains, keys are 0 everywhere. That is the finding: the registry's job is mostly to hold facts that are written down nowhere yet.
• Everything projected validates.

Persist and serve (slice 2)

• Authoring stays in code. A record is registry.py's entry projected onto the v1 shape (registry_reconcile.from_bev), plus the facts registry.py cannot derive, merged from registry_facts.py. That file is empty: nothing is classified until the owner decides it. It may set label, rights.*, trust.*, shape.grains/keys/raw_fields_ref and nothing else; status, kind, acquisition.* and shape.tables stay registry.py's, so there is no second place that can disagree about what runs. An id with no record, or a field it may not set, is an error.
• source_registry table (migration 0061). One row per record: the record as JSON, a content hash, and first_seen_at / changed_at / synced_at / retired_at. Never written by hand or by an API. Its reasons to exist: the master build can join to rights/trust/keys in SQL, and a change to a record leaves a dated trace.
• registry_store.sync() runs at dashboard startup after the migration (and as python -m hoodie_bev.registry_store sync). It validates every record first and refuses on any problem, refuses an empty registry rather than retiring everything, and marks an id that left the code retired_at instead of deleting it (it is revived if it returns). Two deploys syncing at once cannot both insert the same id.
• GET /api/registry/sources[/<id>] serves the stored record with meta and derived beside it. Auth fails closed (see the README). derived is computed from scrape_runs on every read, never stored: last_run, last_success_at, last_success_age_s, cadence_window_s, overdue. overdue means no run that landed data for more than 2 cadence windows (overdue_factor is in every response so nobody guesses). Landed data is stated exactly: a run the source did not flag retry_soon, that is a success (whatever its row count) or a degraded run with total > 0. It is not the dispatcher's rule: that one counts any finished run that is not retry_soon, including failed and no-creds; this is strictly narrower. A degraded run with zero rows (a source blocked on its first page, say) landed nothing and does not make a source fresh. overdue is null — unknown, never false — for a source that has never run, has no cadence window, or whose latest run is no-creds and none ever landed data (an honest skip, shown by the console as its own state, not a stop). A source that has run but never landed data is true. last_success_* stays strictly success and last_completed_* is the data-landing basis, so a consumer can ask for either. The three run lookups are one SQL statement (one snapshot) and use the existing (conn_id, started_at desc) index per source. Every response also carries registry_synced_at (when the registry last matched the code; for a single-source request it is still the registry's, not that row's): if a later sync fails the API keeps serving the last good load, and that timestamp is how a consumer tells. If the stored rows are at a different schema version than the code (a version bump whose sync has not succeeded) the API answers 503 naming both, rather than serve old rows under the new label. (Only active rows count: a retired row is never rewritten, so it keeps the version it was written under, in its own meta.schema_version.)

  Known limits of overdue (read this before relying on it). total does not mean the same thing in every writer: a review that read all 47 record_run callers found rows parsed (most), rows seen, new rows only (salsify), and the size of the whole output table (the master builds). So the predicate cannot be right everywhere: a success with total = 0 counts as landed (right for salsify's caught-up run; wrong for sevenfifty or hemp-scan, whose empty answer is also success); a master build's total is table-sized, so a degraded build that read nothing can still look fresh; and four crawlers set retry_soon on an aborted run that did land some rows, which reads as overdue. last_run (with its total) is served so a consumer can see what the source said. The fix is per-source — a declared landing rule in registry_facts.py, or writers that set retry_soon whenever nothing landed — and is a follow-up, not something to guess at in a shared predicate. Landed row counts are deliberately not served: the console has them, and a count(*) over every table per API call is the wrong cost for a read API.

Authorities: where to look for what a term means

The data-feed registry above says what we PULL FROM. This half says what we CITE: which law or standard defines a word, so an AI answering "what is a reposado?", a data-quality rule checking an ABV, and the taxonomy's basis all point at one record, with the clause. Code: authority_record.py (shape + validator), authorities.py (the authored records), authority_resolve.py (who wins). It sits in the same package and follows the same rules (validate first, unknown is None, versioned envelope), but is its own collection with its own schema_version: rows in sources are scrapeable things, and a consumer pinned to that contract must not start receiving legal instruments.

Decisions (owner, 2026-10-27)

question	decision	why
Where does it live?	Extend the existing registry (same package, store/API/auth pattern), as its own collection.	One registry and one read API for CRM, MDM, the suite and the AI.
First slice?	Tequila, end to end.	Proves the shape, from law to an answer with a citation, before it is scaled.
Two authorities disagree?	Law of the product's origin wins, then the law of the market sold into, then trade convention.	Tequila is what Mexican law says it is, wherever it is sold.

What a record holds

An instrument (Authority): its citation (number, verbatim title, issuer, gazette date), its jurisdiction, the topics it governs, a dated status, the text that was read (url, retrieved_on, sha256 of the exact bytes, and verification: read-primary-text / read-republished-text / read-summary-only / not-read), its terms (name, aliases, clause, a short verbatim excerpt in the original language, and checkable parameters such as min_maturation_months: 2; and parameter_sources, see below), the words it does not define (absent_terms), and defers_to for an instrument that hands the definition to another. A topic carries its origin_jurisdiction; that, not the record, is what makes an instrument origin-law or market-law, because the same regulation is origin law for one product and market law for another.

Rules the validator enforces beyond the usual ones: status in-force/amended needs status_checked_on; a clause number or a quotation needs a text that was actually read (a number from a summary is a claim to check, not a citation); parameters need a text that was read; an absent_term needs the date and a text recorded as complete; an excerpt is at most 240 characters (a pointer, not a copy); a name points at one term (accents and case ignored); every defers_to and topic id must exist.

A value is cited where it is STATED, not where it is filed

A term's parameters are the numbers a rule would enforce, and each number must be traceable to the clause that says it. Usually that is the term's own clause. When it is not, the term declares parameter_sources: {parameter: name of another term of the same record}; the value stays on the term (so a consumer reading parameters loses nothing). The validator requires the named term to carry the same value (same type: True is not 1) among its own parameters and to have its own clause and excerpt, and refuses a self-reference, a chain or cycle of sources, a missing term and two terms with the same name. It does not read the excerpt for the number: authority_verify proves the quotation is in the text, and a person reads the number in the clause. Two cases are recorded for tequila, both surfaced when the suite's UI printed the registry's clause next to the value: the NOM defines Tequila at 4.36 but states the 35-55 % ABV in Table 1 (6.1.1), and 27 CFR 5.148 files Tequila at (b)(1) but states the 40 % minimum in (a), the class rule for agave spirits. The resolver carries each finding's parameter_sources (term, clause, excerpt) and conflicts cite the clause that states each value. That Tequila is subject to the class rule at all is our reading of (a) with (b) (5.148(b) lists Tequila as a type of agave spirits), and the record says so.

Resolution

authority_resolve.resolve(term, authorities=, topics=, market="US", topic=None) is a pure function. It returns the answer and what was passed over: also (other applicable authorities; defers_to_winner where the US rule points at Mexico), not_applicable (a third country's law, with the reason), absent ("the origin's law was read and does not contain the word", dated), conflicts, ambiguous_topics and market_conditions (parameters only the market's law states) and caveats (the answering text was not read in full, the status is not in force, or the instrument is amended).

• Origin is per term, not per instrument. A term names its topic and that topic's origin decides the tier, so a US rule covering both bourbon and tequila is origin law for the first and market law for the second.
• One product at a time. A word two topics define (añejo under tequila and mezcal) with no topic asked for gets no answer and the candidates are listed (a word merely recorded as absent under two products is not "defined under" either).
• Origin law defines, market law adds. When a market instrument states a different value for a parameter the origin also states, both apply where the product is sold, and it is reported as a market-additional conflict. The resolver does not know which of two values is the stricter (a minimum or a maximum?), so the conflict says the stricter binds and leaves that to the consumer. For tequila: Mexico allows 35 % ABV (NOM Table 1), the US class rule needs 40 % (27 CFR 5.148(a)), so resolve("tequila", market="US") carries abv_min_pct 35 vs 40. A consumer that read only the winner's parameters would accept a 38 % bottle on a US label. A disagreement is the same tier (or lower) stating a different value.
• Jurisdictions are ISO 3166-1 countries (249); "EU" is rejected on purpose: supra-national law needs a membership rule this registry does not have.

Tequila, as recorded (read 2026-10-27)

• NOM-006-SCFI-2012 (published in the DOF 13 Dec 2012; in force 11 Feb 2013, labelling 180 days after publication): read from the Secretaría de Economía's consolidated PDF, which carries the DOF's same-day modification inline (seven [Modificación publicada en el DOF el 13 de diciembre de 2012] markers), so it is recorded as read-republished-text: the DOF says its own digitised image is the reference. The platiica listing says Vigente and records a systematic review of 18 May 2023, result CONFIRMACIÓN (the report is a scan and was not read). Every excerpt is verbatim. Classes (4.36.1-4.36.5), categories (5.1.1 "100% de agave", 5.1.2 "Tequila", non-agave sugar at most 49 % and never from another agave), ABV 35-55 % at 20 degrees C (Table 1, 6.1.1).
• The trap, pinned by a test: the NOM's international labels (5.2.2) are Silver = blanco, Gold = joven, Aged = reposado, Extra aged = añejo, Ultra aged = extra añejo. Read "Extra aged" as extra añejo and every class above blanco is off by one.
• 27 CFR 5.148 (eCFR, issue date 2026-10-23): read in full. The US does not define tequila; it is "a distinctive product of Mexico" made "in compliance with the laws and regulations of Mexico governing the manufacture of Tequila". The 40 % minimum is the agave spirits class rule and adds a condition (above).
• Declaration of the Denominación de Origen (1977): read in full from WIPO's PDF (articles 1-6, three transitory articles). Art. 2 ties the name to the "Norma Oficial de Calidad para Tequila", which is what makes the NOM the operative definition; Art. 3 sets the territory as all of Jalisco plus named municipalities of Guanajuato, Michoacán, Nayarit and Tamaulipas. No municipality name is recorded: WIPO's transcription has errors in them ("Pénamo", "Nuevo Parangarigules"). WIPO lists two later modifications that were not read, so the status is amended with what they changed unknown.
• "cristalino" and "mixto" are not in the NOM. They are trade words; no age or process rule attaches to them. Recorded as absent_terms, not left blank.

Re-checking a record

python -m hoodie_bev.authority_verify <authority-id> --text extracted.txt --source downloaded.pdf fails if an excerpt is not verbatim in the text, a word recorded as absent is present, or the source's sha-256 / size no longer matches. It proves a quotation is in the text and an absent word is not (accent-, case- and hyphenation-insensitive); it does not check clause numbers, parameters, aliases or parents, and it does not tie --text to --source, so a person reads those and extracts the text from the same file. Run it when a record is written and whenever the source may have moved. (The first review caught a record that misdescribed what had been read; this is the check that does not depend on anyone's memory.)

Persist and serve

The same pattern as the data-feed registry's slice 2, in its own table and routes.

• Authoring stays in code (authorities.py); authority_store.sync() makes the authority_registry table (migration 0062; one row per authority and per topic, keyed (kind, id)) match it. It validates the whole set first (each record and the references between them), refuses an empty load, never deletes (retired_at, revived if it returns), and locks every existing row up front so overlapping syncs serialise: two code versions syncing at once each retire the other's row, and sorting only the upserts left that step locking in the opposite order, which Postgres resolved by aborting one (reproduced in review, now a test). A failed sync rolls the caller's connection back and writes nothing. The dashboard runs it after the sources sync, in its own try, so neither failure can hide or cause the other. Nothing else writes the table.
• GET /api/registry/authorities[/<id>] serves the stored records with meta beside them; GET /api/registry/authorities/resolve?term=&market=&topic= runs authority_resolve.resolve over them and answers with the citation, what was passed over, the conflicts, the market-only conditions and the caveats. Missing term, an invalid market or an unknown topic is 400. resolve is a route, so no authority may use that id.
• Same gate as the sources API: one REGISTRY_API_TOKEN, fail closed (503 if unset, checked before any database access, 401 for a bad token), Cache-Control: no-store. The authority contract has its own schema_version: ?schema_version=N the registry is not at is 409; stored active rows at another version are 503.
• The resolver never runs on rows this code cannot vouch for. hydrate() rebuilds stored rows with a strict loader (an unknown field is an error, not a silent drop) and re-validates them as a set (including a row of the wrong type that makes the validator itself raise); if that fails, /resolve is 503, not a best guess. The plain list and by-id routes serve stored rows as stored, and are not validated: the refusal is /resolve's. Retired rows never answer, whatever include_retired says.
• Tests run against a real Postgres (tests/test_authority_store_db.py: migration, add/unchanged/changed/retire/revive, an empty or invalid load that changes nothing, four syncs at once, a missing table on both connection modes, and the HTTP route end to end), plus pure tests for the store, the API and the startup order.

Not done, on purpose

• Not read: the Declaration's two later modifications (WIPO MX090, MX092), the 2023 systematic-review report (a scan), and the DOF for changes in the last weeks (status was checked against the Secretaría de Economía listing and searches, not day by day).
• Not deployed by this change. Merging ships nothing; the dashboard service is deployed deliberately (scripts/deploy_service.sh), and its startup then runs migration 0062 and the first sync.
• Not consumed yet: product_taxonomy.py's BASIS strings are still free text and US-only; pointing them at authority ids, and a data-quality rule that checks an ABV or an age statement against parameters (remembering the market-additional conflicts above), are consumers to build once this is served.
• Other products (mezcal and its NOM, cognac, scotch, bourbon, wine) are not recorded. Each needs the same reading, not a guess from memory.

Next slices

1. ~~Persist + serve.~~ Done in slice 2 (above).
2. Point the dispatcher at the record for the fields that already mean the same (cadence, lane, where), with registry.py as a generated view during the move.
3. Consumers. (Needs REGISTRY_API_TOKEN set on the dashboard service by the owner; nothing consumes the API yet.) CRM: read shape and rights to label what a number came from. Master build: read trust and shape.keys instead of hard-coding which source may seed identity.
4. Populate rights, trust, keys, source by source, starting with the ones that already carry a verbatim rights record.

Open decisions (not mine to make)

• Trust classification. master_role values here are proposed from the standing rules (TTB tier-0, VIP corroborating). Each source's role needs the owner's sign-off before it is written down.
• Rights. facts_use for a source is a legal classification. The suite has a single rights record today (dam-bacardi); every other source needs review, and anything that needs_counsel is not mine to settle.
• Which side is current for each of the 14 disagreements.
• Free-first placement. The report only flags klass=mac against where=cloud. A suite cost_class of proxy/bd has no counterpart here (this repo records none), so e.g. walmart (suite: proxy) runs at where=cloud here without showing up as a disagreement. That is a gap in what this repo records, not an agreement.
• cost_class="mac" names a place, not a price. v2 should split where it runs from what it costs; carried verbatim in v1.