The agent that is not allowed to see the money
I built a tariff reclassification agent without access to duty rates. The frozen August evaluation reached 88.9% at eight digits on its DEAD_CODE stratum; this update separates that result from the six-digit baseline and the production triage path.
Contents
I built this for the All Things Agentic Hackathon, and this post is my entry for the content bonus.
Updated October 8, 2026: I corrected the description of the evaluation path and separated the HS8 results from the HS6 baseline. The figures below are from the frozen August 30 run, not a new model run.
The US tariff schedule shipped three revisions in three weeks this summer: Rev 14 on July 29, Rev 15 on August 3, Rev 16 on August 11. Every one of them retires some codes and splits others. Somewhere in a manufacturer's catalog, a line that cleared customs last month is now filed under a heading that does not exist.
A dead code usually has more than one legal successor. At eight digits those successors can carry different duty rates, and the difference on a single entry runs into thousands of dollars. Customs does not tell you which one you picked wrong. The penalty notice does, later, with interest.
So the obvious build is an agent that reads the catalog and re-classifies it. I built something narrower than that, and the narrowing is the whole point.
Most of the work does not need a model
Production triage and the evaluation are different paths
The official correlation table already says what happened to each code. The production triage settles an unchanged code with no scope warning, or an unambiguous one-to-one successor, by program. A surviving code can still need judgment: when its scope changed or the correlation splits, the triage routes it to the agent. Those rules are in the triage implementation at the evaluation commit.
The frozen evaluation did not measure that routing shortcut. All 500 saved answers have agent tool calls, including the 90 SCOPE_REVIEW cases and the 122 SURVIVED cases. The 212 cases in those two strata were not 212 cases that bypassed the agent. The other 288 are labeled DEAD_CODE by the evaluation builder. The saved run makes the agent calls checkable.
Those labels come from the HS2017-to-HS2022 correlation, not a check that every prior code has disappeared from the current schedule. The builder labels a case DEAD_CODE when its target HS6 is not an HS2017 source key but is present in the reverse correlation, then chooses a prior code from that reverse mapping. If the target is an HS2017 source key, it assigns SURVIVED for a one-to-one mapping without an ex scope flag, or SCOPE_REVIEW otherwise. This is the rule in the sampling implementation; the README's shorthand that the prior HS6 no longer exists is broader than that rule establishes.
Compare results at the same number of digits
The agent scored 92.8% at eight digits across all 500. That blended result includes cases whose old and correct codes already share their first six digits. Splitting the strata shows what changed, and keeping the digit level explicit avoids comparing two different tasks:
| evaluation stratum | n | agent correct at HS8 | agent correct at HS6 | keep-or-map baseline at HS6 |
|---|---|---|---|---|
| DEAD_CODE | 288 | 256/288, 88.9% | 269/288, 93.4% | 21.2% |
| SCOPE_REVIEW | 90 | 86/90, 95.6% | 88/90, 97.8% | 97.8% |
| SURVIVED | 122 | 122/122, 100% | 122/122, 100% | 100% |
The 21.2% mechanical floor is a six-digit result. Its comparable agent result is 93.4%, not 88.9%. The eight-digit result tests the more specific US tariff line, where duty rates attach. The two other strata primarily test that finer selection rather than a change of HS6. These are the numbers in the frozen score report.
The report calls the mechanical policy “carry forward,” but it does not mean keeping the old number unchanged in every case. It keeps the prior HS6 if that code exists in the snapshot; otherwise it selects the correlation table's first candidate. The baseline implementation defines that keep-or-map policy. The separate policy that always takes the first candidate scores 25.3% on DEAD_CODE, not 21.2%. Neither policy represents a customs professional's normal practice.
Method note: what the frozen set can establish
The set contains 500 CBP rulings issued on or after January 1, 2022, drawn on August 17, 2026. The CBP-assigned code is the answer key. The input combines a prior code recovered from the HS2017-to-HS2022 correlation with the ruling's merchandise description. Eligibility was checked before sampling, and the precedent index excluded the 500 answer rulings and their related rulings. Description extraction follows the NY-letter template, so HQ rulings and revocation notices are under-represented. The sampling and exclusion rules are documented in the evaluation README.
This is a historical classification benchmark on that selected population. It does not isolate changes introduced by the three August schedule revisions, measure accuracy on a manufacturer's live catalog, measure the share of production cases that avoid model calls, or establish acceptance of a customs filing. In the score report, “shipped” means the answer cleared the program gates, not that customs accepted it. This update checks the artifacts at commit 00cc8d71bf0a14072e0a3013803220501d4afd98; I have not rerun the agent or replaced the competition model.
The agent cannot see duty rates
The classification agent gets four read-only corpora: chapter and section notes, tariff lines, ruling search, and ruling texts. It does not get duty rates. It does not get chapter 99, where the additional measures live. It has no tool that returns money.
That is a deliberate hole in its capabilities, and it exists because of a failure mode nobody can audit after the fact. If a model can see that one candidate code carries 8% and the other carries 32%, and it is asked to pick one, you can never prove afterwards which consideration produced the answer. Its own explanation is not evidence: post-hoc rationales are the one thing language models are unconditionally good at.
Removing the rates from its tools removes the live price comparison from the decision path. It does not prove the model has no tariff knowledge from training. The duty is computed afterwards by a program, from the code the agent chose, and shown to the person who signs.
What the citation checker actually checks
The agent returns citations: a chapter note, a tariff line, a past ruling. A deterministic checker resolves the references against the frozen snapshot. For chapter notes and tariff lines, it also compares the quoted words with the stored text, allowing changes in punctuation, case, line breaks, and footnote markers.
Ruling quotations have a different boundary. The index holds ruling numbers and subjects, not the full ruling text, so the checker verifies that a ruling exists but reports its quotation as not checkable. It does not reject an answer merely because that ruling quotation was not checked. That limitation is explicit in the citation verifier.
Anything that does not resolve stops the line. On the 500-item run, seven answers cited something that did not hold up: three quoted words not found in the source, three referenced an authority with no row in the current schedule, and one used a note reference that could not be resolved. None of those seven cleared the citation gate. The checker can verify that a quotation exists; it cannot establish that the legal argument or selected code is correct.
The interesting question about a checker is not whether it passes the things that are right. It is whether it catches the things that are wrong, and you cannot demonstrate that by waiting for a hallucination to happen on camera. So I took 423 answers with an eligible chapter-note or tariff-line quotation that had already passed, inserted one word into one real quote, and sent them back through the same checker used by the application. The fault-injection implementation excludes ruling quotations.
423 altered answers. 423 caught. Zero missed.
The inserted word is solely, which reads like a qualifier a paraphraser would add rather than obvious damage. The test caught that mutation in all 423 cases. It tests quotation integrity under this fault model, not every way a citation or classification can be wrong.
Three things that broke
A wrong filename returns HTTP 200 with an empty body
The government endpoints this reads from do not 404 on a bad file name; they return success and nothing. For the screening list, an empty response means every party passes screening, which is the most dangerous failure available in this domain. Status codes are never sufficient. The health gate checks byte and row floors and re-verifies the sha256 of what landed on disk.
The evaluation set leaked its own answers
CBP ruling texts open with sentences like "you request a ruling on the applicability of subheading 6201.93". The merchandise description I was feeding in as input carried the answer inside it. Cutting each text before its first tariff reference would have destroyed 23% of the items, some down to 8% of their original length, because the reference often comes before the product description. Redacting in place instead kept every product fact. A gate re-verifies after each rebuild that no description can read its own answer; it caught three regex mistakes, one of which had taken leakage from zero items to 58.
Deploying it killed it
The precedent index holds 218,606 rulings. At 1 GiB the Cloud Run container was terminated mid-batch at 1030 MiB, and every case the agent had open came back stranded. The memory limit is a load-bearing setting, and the deploy script now says so in a comment, next to --no-cpu-throttling (background threads keep working after the response is sent) and --max-instances=1 (in-process worker state).
What it refuses
When the description does not carry the one fact the decision turns on, the agent does not pick the likelier candidate. It refuses, names the missing property, and names the department that owns the answer. The case waits in Cloud SQL with its history intact, and when a person answers days later, the re-run cites the answered fact rather than the guess.
It does this on 6 of 500 items. A refusal rate of 1.2%, against a design cap of 10%. A system that escalates whenever it is unsure hands its whole caseload back to the person it was supposed to help; the cap is what keeps "refuses to guess" from becoming "refuses to work".
The shape
A table settles what a table can settle. One agent argues what needs arguing. A checker verifies references and the source text it can compare. A person signs once.
The code is at github.com/cyh7789/tariff-reclassification-line. The competition build ran on Cloud Run with Cloud SQL, and the agent was Gemini 3.7 Flash through the Google GenAI SDK on Vertex AI.