Adaptive review — the next card is the one that teaches the most

Status: phase 1 built (adaptive_review.py, migration 0102, registry adaptive-review; §10 has what it measured). Phases 2-3 are design. Owner's idea (2026-11-10): "a neural network constantly feeds outlet check sets over and over, anyone can just match and teach it ... mimic those tests where the next question is based on your prior performance."

That is two known techniques at once:

1. Active learning — a model chooses the pair whose answer would change it most, instead of a fixed per-stratum draw.
2. Computerized adaptive testing (the GRE's method, item response theory) — each reviewer has a measured accuracy and each pair a difficulty; a new reviewer is shown pairs whose answer is known until we know how good they are, a proven one is shown the hard pairs.

Together they let more people answer outlet pairs safely, and spend every answer where the master is least sure.

1. Where we are (measured 2026-11-10, production, read-only)

Person answers on outlet pairs	878 — 583 same, 128 different, 167 unsure (711 decided)
Answers per day	bursty: 447 on 10-30, then 0–174 a day
Reviewers	one — match_candidates.verdict holds one answer per pair, with no reviewer recorded; writes are gated by one shared DASHBOARD_TOKEN
Candidate pairs in the latest outlet-pairs build	1,207,689 (same-address 676,065 · near 336,427 · control 170,814 · proximity 24,383)
Claude answers on them	Haiku v7: 368,015 different / 70,007 same / 2,492 unsure; Opus v7: 53,856 same (it only sees Haiku's same + unsure)
Signals stored per pair	name_sim, address_sim, jw, km, unit, legal_name, phone (+ stratum and source pair)
Fixed sets drawn for a person	calibration 200, judge-check 200, agent-check 134, haiku-review 200, gaps 204

So: ~700 decided answers against 1.2 million open questions, from one person. Every set so far was drawn by a rule we wrote (a stratum, a cap, a bucket). None was drawn by asking "which answer would we learn most from?".

2. What it looks like

A new chip on the Match page, Next best (outlets only), next to Easiest first / Calibration set / Gaps set / ... It serves one card at a time from all 1.2M candidates. After each answer the model refits (seconds) and the next card is chosen again.

What a reviewer sees does not change: two records, Y / N / U, blind (no score, no Claude answer, no stratum on the card — the existing rule). After answering they see Claude's answer and the outlet(s) as today. What changes is which card comes next, and that their answer carries a weight that depends on how often they have been right.

3. The model — small, on purpose

Not a neural network. With 711 decided labels a neural net would overfit and could not say why it asked. The large neural network in this loop is already Claude; the piece that is missing is a small one that learns from people. v1 is a logistic regression over:

• the stored signals: name_sim, jw, address_sim, log(km) with a missing-pin flag, unit relation, legal_name, phone;
• the stratum and the two sources (one-hot; rare source pairs pooled);
• Claude: Haiku's and Opus's verdict + confidence on the pair's current records (input_hash must match, as everywhere);
• what the master did: the rule tier that applied or held the link, if any.

Pure Python (no numpy / pandas in the image — CLAUDE.md), ~30 features × ~1,000 rows: a fit is milliseconds. Coefficients are printed by --report, so "it thinks pin distance matters more than the street" is readable. A gradient-boosted model is v2, only if the logistic one plateaus on the measure in §7.

What the model may do: order the queue. Nothing else. It never merges, never splits, never becomes a label — the same standing as Overture's fetch hints ("only ORDERS the crawl"). A model that wants to merge becomes a tier and earns a gate like the judge did, in a later, separate design.

4. Choosing the next card

Score every candidate with p = P(same) from the model, then rank by value of an answer:

• uncertainty — 1 − |2p − 1| (highest at p = 0.5);
• disagreement — how many of {model, Haiku, Opus, the master's rule} disagree (query-by-committee); a Haiku 'same' that Opus called 'different' is exactly the pair the screen and the confirm stage need a person for;
• consequence — would it change the master? A pair the Claude tier merges or holds (the master's decision depends on it) counts more than a pair both sides already agree to leave apart; far-apart and no-alcohol pairs are never served (the existing exclusions).

value = uncertainty × (1 + disagreement) × (1 + consequence), with diversity: at most one card per outlet and a cap per source pair in any window of 50, so one confusing mall cannot take the whole session.

Every card is one of three kinds, mixed in a fixed ratio:

kind	share	purpose
chosen	~80%	highest value, as above
random	~10%	a uniform draw from the candidates — keeps an unbiased check on the model (§7) and stops it only seeing its own blind spots
known	~10%	a pair whose answer is already settled (§5) — measures the reviewer

The reviewer cannot tell the three apart (one neutral stratum, adaptive, as the Gaps / Agent sets already do).

5. Reviewers — "anyone can match" safely

Today there is one person and one shared token. Opening it up needs:

• A reviewer per answer. New table match_label (pair_id, reviewer, verdict, note, served_as chosen|random|known, model_p, at), one row per reviewer per pair, never overwritten (a changed mind is a new row; Undo deletes the reviewer's own latest row only). match_candidates.verdict stays as it is — the owner's answer — so every existing gate, refusal and report keeps reading what it reads today.
• A token per reviewer, issued by the owner (reviewer_token: handle, hashed token, created, revoked). A handle, not a name or email. This is a new write path, so it gets the scrutiny CLAUDE.md asks of every write path: tokens are revocable, only /match/label accepts them (not Run now, not builds), and a revoked reviewer's answers stay recorded but drop to weight 0.
• Known-answer pairs (the gold): decided answers where the owner and the pinned Opus judge agree, on the pair's current records (how many of the 711 qualify: 292, measured 2026-11-10). More are added as the consensus grows. A reviewer's known cards never include a pair they answered before.
• Measured accuracy per reviewer: correct / decided on known cards, with a Wilson 95% interval (the repo's standard), split into 'same'-accuracy and 'different'-accuracy because the two errors cost differently (a false 'same' is a false merge).
• Levels, like the adaptive test:
  • new — known cards and easy chosen cards (p far from 0.5) until 20 known cards are decided;
  • trusted — lower bound ≥ 0.90 on both kinds: full chosen queue, hard cards first;
  • paused — lower bound < 0.75: only known cards again; their other answers stay recorded at weight 0.

  The owner starts trusted, but is measured the same way (rechecks have already changed 5 of the owner's answers — nobody is exempt).

• Combining answers (Dawid–Skene, simplest form): each decided answer contributes log-odds weighted by that reviewer's measured accuracy; 'unsure' contributes nothing. A pair's consensus is decided once the combined odds pass 20:1 and no trusted reviewer disagrees; otherwise it goes to a second reviewer (a disagreement is itself high value).

6. What answers are used for, and what they must never feed

• The model trains on every weighted consensus answer, whatever kind of card it came from.
• The gates do not change. The judge gate, the agent gate, the Gaps and duplicate precision all keep reading their own random or stratified draws. An adaptive queue serves hard pairs on purpose, so accuracy measured on it is biased low — the same reason the calibration set exists (the "Easiest first" queue biased it high). A test will pin that no gate reads served_as = 'chosen' answers.
• Refusals (un-merge) stay the owner's 'different' in v1. In v2, a trusted-consensus 'different' may create a refusal too — that is the point of more reviewers, and it is decided after v1's numbers (§7) are in, not now.
• Rechecks (label_audit) keep working on the owner's answers; for other reviewers a known card answered wrong is the recheck.

7. How we know it works

Hold out the random cards (§4) as the test set. As answers accumulate, plot the model's log-loss and accuracy on the random cards against the number of answers, and compare with what the same number of answers drawn by the existing strata would have given (refit on those labels only). Active learning is worth keeping only if it reaches the same accuracy with clearly fewer answers. The report also gives, per reviewer, known-card accuracy with its interval, and how many candidate pairs the model now calls confidently (p < 0.05 or > 0.95) — the share of the 1.2M that no longer needs anyone.

If after ~500 adaptive answers the curve is no better than the stratified one, stop and say so: the strata were already doing the job.

8. Phases

1. Model + Next best, owner only. adaptive_review.py (fit, score, choose, report), the chip, match_label written alongside match_candidates.verdict. Done when the queue serves, refits after each answer, and --report shows the learning curve on random cards. Migration: match_label. No new reviewers yet.
2. Reviewers. reviewer_token, per-reviewer levels, known cards, consensus, a reviewer page (their accuracy, their level — shown to them). Done when a second person can answer with their own token and their known-card accuracy is on the report.
3. Consensus refusals (decided on phase-1/2 numbers): trusted consensus may split a merge.

Items (kind = 'item') can follow the same path later; outlets first because that is where the gates, the judge and the candidate table are.

9. Questions for the owner

• Who are the first other reviewers (how many, how well do they know the stores)? It decides how strict trusted needs to be.
• Is 80 / 10 / 10 the right mix to start? More random cards make the measurement faster but teach the model less per answer.
• Should a reviewer see their own accuracy and level? (Proposed: yes — it is what makes the test feel fair.)

10. Phase 1 as built (2026-11-10)

What differs from the sketch above, and why:

• Cards are served 9 chosen : 1 random. The known cards are phase 2: the only reviewer so far answered them.
• The model refits after every answer on an adaptive card, in a background thread of the dashboard (one refresh at a time; answers that arrive meanwhile ask for one more), so the answer returns at once. A page loads the 10 open cards; the next page load gets the re-ranked queue.
• The pool, not 1.2M pairs, is re-scored per answer. The daily build (after outlet-pairs) scores every eligible pair and keeps the top 3,000 by value plus a uniform 1,000; each answer re-scores those 4,000. A card queued more than a day ago and still open is retired by the next build.
• The master is NOT a feature (§3 listed "what the master did"): a person's 'different' splits a merge, so it would read the label back. It only raises a pair's value (consequence).
• An adaptive card is always a NEW Match pair under stratum adaptive. A pair the Match page already shows is skipped, not borrowed. So every adaptive answer sits under a stratum no gate reads, and outlet_judge --set answered (scoring a new judge) excludes it — a test pins both.
• Every answer from any queue is now also a match_label row (who = 'owner' for now, which queue served it, the model's guess); Undo deletes the latest row and puts an adaptive card back in the queue.

Measured read-only on production the same day (the module run against the live tables, nothing written):

Decided answers the model learns from	711 (583 same)
Fit time	0.06 s
5-fold cross-validated accuracy	96.1% (log-loss 0.136) — always guessing 'same' gets 82.0%
For comparison, on the same answers	Opus alone 98.3% of the 297 it answered; Haiku alone 96.3% of 273
Eligible candidate pairs	998,241 (209,090 skipped: a no-alcohol outlet on one side)
Pairs the model is already confident on (p < 0.05 or > 0.95)	23.5% — three quarters of the candidates are still open questions
Full build (scan + score)	40 s

Heaviest weights: Opus 'different' (−2.9), the scorer's score (+2.7), address similarity (+2.1), pin distance (−1.7 per log-km), Opus 'same' (+1.6), a differing suite (−1.4), a Texas licence on one side (−1.1). Phone and legal name carry no weight yet: too few answered pairs have them.

Read this before trusting the 96%. The answers come mostly from strata unlike the pool the model now ranks: addr-agree, dup-listing, agent-check and the Gaps set are mostly 'same', while the pool is mostly near / same-address / control pairs that are mostly different. Cross-validation on the old answers says the model fits them; only the random cards measure it on the pool. Until 20 are answered, --report says UNMEASURED.

Run it: python -m hoodie_bev.adaptive_review --build --apply (also daily), then Next best on the Match page; python -m hoodie_bev.adaptive_review --report for the weights, cross-validation and — once 20 random cards are answered — the learning curve.

11. The web agent on every card (2026-11-11)

The owner's own check is to search both addresses and read the records. outlet_agent.run_next does that for each open Next best card (set next-best, Haiku then Opus, capped at AGENT_NEXT_DAILY_USD per day) while it waits. Its answer is a model feature and a vote in the disagreement term, and it is shown only after the person answers, with its sources. It never merges (the agent tier reads held-1 only). --next-report measures it against the person. If the random cards show high agreement, the next step is phase 3's "confident cards skip the person, a random share still goes to them". That decision belongs to the owner.

12. Attribute tags (2026-11-11)

Each record on a Next best card has three dropdowns: premise, trade channel and chain type (outlet_tags.py). They use the master's own vocabulary, start blank, and save per record. outlet_tags --report scores the master's channel and chain rules against them. Next, once there are enough tags: give a card more value when the master's attribute for its records is unknown, conflicting (channel_conflict) or often wrong in the report, so tagging teaches the attribute rules the way answering teaches the match model.

13. Web context on the card (2026-11-11)

A context task per Next best card (Haiku, never asked the question) finds cited facts about the records and their streets: renames, moves, name changes, store-in-store. The card shows them BEFORE the answer, with no verdict. Facts without a page the agent found, and facts that state a verdict, are dropped. Trade-off: the person and the pair agent now share facts, so their agreement is partly correlated.

14. Readable context, and time to match (2026-11-11)

Context bullets are labelled, rewritten in at most 20 words, tagged A/B/both and grouped on the card. Each answer stores ACTIVE seconds on the card (tab visible, input within 2 minutes; an idle check pauses the clock and marks the answer away) and what context the card showed. Comparing median time with facts against nothing / pending is the first read of whether context saves time.