tax-doc-classifier

How we reviewed this: we ran the classifier in Docker on Node 24 with poppler and a real Jev key, against IRS form PDFs we downloaded from irs.gov, a page of the 1040 instruction booklet, a California Form 540, and a page of ordinary prose. We also ran the test suite, the type check, and installed the package the two ways the README describes.

What it does

tax-doc-classifier, from Kyoto (heykyoto.com, “an AI Financial Advisor grounded in your tax liability”), takes the text of one page of a tax document and says which IRS form it belongs to and what kind of page it is. There is no trained model and nothing to host: the whole classifier is data/criteria.json, a generated description of 261 forms, plus one Jev request that picks among them.

One request carries two Choice questions over the same page: kind (form page, instructions, blank, cover sheet, state form, broker statement, other) and form (230 options, plus not_in_this_list). Five corporate and foreign form families — 5471, 8865, 8933, 1118, 5713 — absorb their 35 schedules and get a second, small question only when the first lands on the family. Blank pages are answered in code with no call at all.

The criteria file is generated, not written: pnpm build-criteria reads the IRS’s accepted-forms list, downloads each PDF, and extracts the printed label, title, page count, parent, box labels, and the sibling forms a page must not be confused with. Re-run it when the IRS publishes new revisions. IRS works aren’t copyrighted, and the repository says so in a separate data licence.

The README leads with cost and speed comparisons against the LLM pipeline this replaced, and reports its own evaluation results on two corpora. Consistent with our policy, we don’t republish those figures; they are the author’s, measured on their machines.

Using it

The tests and the type check passed. Then we pointed it at real pages:

Page	Answered	Confidence
Form 1040, page 1	form-1040	gate cleared
Form 1040, page 2	form-1040	gate cleared
Schedule A (Form 1040)	form-1040-schedule-a	gate cleared
Form 8880	form-8880	gate cleared
Form W-2, page 1	form-w-2	gate cleared
Schedule J (Form 5471)	form-5471-schedule-j	gate cleared, two calls

Six pages is not an evaluation, and we make no claim about accuracy at scale. What it does show is that the pipeline works end to end as documented, including the two-step path for the 5471 family and the gate at 0.95.

The trap: read gated, not form

The interesting results were the pages that don’t belong at all.

We fed it page 1 of a California Form 540 — a state return, deliberately outside the federal list. The kind answer was right (state_tax_form), and not_in_this_list took essentially all of the form probability, exactly as designed. But the returned result.form was form-982, an unrelated federal form, with a confidence of 0.000 and gated: false.

A page of our own newsletter prose came back the same way: not_in_this_list took the probability, kind was letter_or_other, and result.form was form-8082.

This is deliberate in the code — the winner is picked excluding not_in_this_list, so form is never that value — and the confidence and gated flag both tell you not to trust it. But the shape invites a mistake: result.form always reads like an answer. Log it, store it, or show it to a reviewer without checking gated first, and a California return is filed under Form 982. If you use this library, gate on gated (or formConfidence) before you so much as look at form, and treat not_in_this_list winning as its own outcome.

One smaller thing: kind has no gate. Our instruction-booklet page came back as a form page rather than instructions, at a confidence the form gate would have rejected. The kindConfidence is right there in the result; the README’s gate advice is about the form.

Two packaging bugs

Both of the README’s own install paths fail as published. We reproduced each in a clean container:

1. The documented import doesn’t resolve. import criteria from 'tax-doc-classifier/data/criteria.json' — the line in the README’s first code block — fails with ERR_PACKAGE_PATH_NOT_EXPORTED, because package.json only exports .. The file ships in the tarball; it just isn’t reachable.
2. Installing from GitHub gives you no code. The README says pnpm add github:<org>/tax-doc-classifier. dist/ is gitignored and there is no prepare script, so the install lands a package with data and docs and no build output, and importing it fails with ERR_MODULE_NOT_FOUND.

Both were a few lines to fix, and the author fixed them in 3e95a77 the day we reported them: exports now includes the criteria file, and a prepare script builds dist/ on a git install. We checked it ourselves afterwards, installing that commit from GitHub into an empty project in a clean container: dist/ is built, the main import resolves, and tax-doc-classifier/data/criteria.json loads with all 261 forms. Note that the package is still not on npm, so a git install is the only route.

Permissions and data

• Page text goes to TypeSafe. The state sent per page is a 12-line header, up to 2,500 characters of body, and a 6-line footer. On a filled return that includes names, addresses, taxpayer identification numbers and amounts. This is a classifier for tax documents, so treat the data path as the main decision: check TypeSafe’s terms and your own obligations before sending client returns, and redact first if you need to.
• Your key is read from the environment. TYPESAFE_API_KEY, never written to disk by the library, and .env is gitignored.
• Retries are broad. The backend retries up to four times on any failure, including a 401 or a 400 that will never succeed, and including a timeout after a long wait — where the request may well have been processed already. Narrow that if you care about your bill or your error latency.
• No IRS PDFs are redistributed. The evaluation downloads them from irs.gov at run time. The bench corpus (TaxCalcBench) is a separate project and isn’t included.

Maintenance

Two commits on September 18, Apache-2.0, a separate data licence for the IRS-derived files, a small test suite covering the id grammar, and a reproducible evaluation harness. Open sourced out of a working product rather than built as a demo, which shows in the parts that matter: generated criteria, an explicit gate, a documented fallback, and a Backend interface with exactly one method if you want to put something else behind it.

Verdict

This reads like production code rather than a demo: one request per page, an explicit confidence gate, a fallback for everything under it, and a form registry generated from the source of truth instead of hand-maintained. On the pages we tried it did exactly what it says.

Two things to do before you build on it. Fix the packaging, or vendor the source until upstream does. And write your own code as if result.form were unset whenever gated is false — because for anything that isn’t a federal form, the name it gives you is not an answer.

For other document work, see doc-router in our projects list, and our Best Jev Tools roundup for the tools we recommend.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Sep 20, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.