Research
What does an LLM need to classify a transaction?
A directed study at BYU with Prof. Mark Keith. When a deployed classifier gets an accounting transaction wrong, should you buy a better model, give it more information, or send it to a human? This benchmark measures which inputs actually matter, and whether more context can stand in for a bigger model.
- Advisor
- Prof. Mark Keith, BYU
- Status
- Design complete · data prep in progress
- Data
- Real transactions from 3 businesses, de-identified
- Write-up
- Coming soon
Question
Transaction categorization is usually treated as text classification: description in, category out. But a bank feed carries more than a description: the amount, the date, whether it recurs, and how the business booked that counterparty before. Transactions also differ in kind. A software subscription, a loan payment, and an owner draw need different information to classify correctly.
The study asks which of those inputs an LLM needs, how that changes by transaction type, and how input richness trades off against model capability and label granularity.
Design
Every test transaction is classified under every condition, so item difficulty is held constant and only the factor being studied changes.
- Five cumulative input levels: description only → + amount → + date and recurrence → + retrieved history (10 similar prior transactions from that client, with their labels) → + industry and account type
- Three model tiers (small, mid, frontier) spanning roughly 10× in price
- Two label granularities: 35 accounts built on IRS Schedule C, and 6 transaction cycles
- Items sampled across six transaction cycles and three “evidence” levels, from easy repeat vendors to brand-new counterparties
- Two repeats per condition, about 21,600 classifications in total
Pipeline
Hypotheses
- A client’s own history contributes more than any other single input, because the right category is often a house convention, not a fact about the merchant
- More input can substitute for model capability: a small model with full context matches a frontier model given only the description, at a fraction of the cost
- Which inputs matter depends on the transaction type. Expenses lean on history; financing and transfers lean on amount and account structure
- Accuracy drops at fine granularity, mostly where the answer depends on information outside the transaction line
- Stored production labels agree with independent expert labels less often than assumed
Hard parts and decisions
Trustworthy ground truth
Production labels can’t be the answer key: many were approved with the model’s own suggestion on screen. Instead, an accountant labels every test item cold, seeing only the raw transaction. A second accountant labels a 60-item overlap independently, and agreement is reported as Cohen’s κ.
No peeking at the future
Retrieved history is bounded to transactions dated before each test item, and the code checks every neighbor and fails loudly on a violation. Without that, history would look more useful than it really is.
De-identification you can measure
Bank descriptions name real people: check payees, Zelle and Venmo counterparties, memo notes. The export pipeline replaces them with stable pseudonyms and reports exactly what it scrubbed, so privacy is measured rather than assumed. No data reaches a model API until that check and the advisor’s written approval are both in place.
A baseline that keeps the LLM honest
A simple nearest-neighbor approach (copy the label of the most similar past transaction) turned out to be strong on easy items. I’ve proposed adding it as a baseline, so the real question becomes whether an LLM adds anything on the hard cases.
Status
The harness is built and tested: export, de-identification, tagging, retrieval, and a model runner. A 200-item pilot (12,000 planned calls) has been dry-run end to end. Live runs start once the dataset approval is in writing. The chart below shows the shape of the main result, using placeholder numbers until then.
Accuracy by input level and model tier
Placeholder data- Small model
- Mid model
- Frontier model
D1 Description onlyD2 + AmountD3 + Date & recurrenceD4 + Retrieved historyD5 + Industry & account
View as table
| Model | D1 | D2 | D3 | D4 | D5 |
|---|---|---|---|---|---|
| Small model | 55% | 59% | 61% | 78% | 80% |
| Mid model | 62% | 66% | 68% | 82% | 84% |
| Frontier model | 68% | 71% | 73% | 85% | 87% |