95% accurate still leaves ten thousand products in the wrong place

A bottle of barista oat drink arrives in a vendor feed. The vendor has filed it under their own “Dairy” heading, which is where it sits in their shop and is wrong in this one. The title says nothing about milk. The ingredients start with water and oats. The brand is one the site has never stocked.

Somewhere in an eight-level taxonomy there is exactly one right place for that bottle, and several dozen plausible wrong ones.

Now do it two hundred thousand times, and keep doing it every time a supplier pushes an update.

Where the catalogue actually broke

The client is a UK online grocer. Their products come from supermarkets, wholesalers and specialty retailers, and each of those sources arrives with its own naming conventions, its own attribute fields and its own source category structure. That last one is the trap. A vendor category looks like a helpful hint and is in fact a label from somebody else’s taxonomy, built for somebody else’s shop, with different departments and a different idea of what belongs next to what.

So products landed in the wrong place regularly. Discovery broke, navigation broke, and shoppers ended up filtering through noise to find things the site definitely stocked.

The fix, for a long time, was people. New items got hand-reviewed and reassigned. That works at a few thousand SKUs. Past a certain size the review team stops being a quality gate and becomes the rate-limiter on how fast new inventory can go live at all, and vendor pushes that ought to reach the site in minutes were taking days.

There was a second failure underneath, quieter and more corrosive. The taxonomy had been redesigned to eight levels, which is a serious piece of work and a genuinely good one. But mapping was left to individual judgement, and individual judgement varies — between vendors, between analysts, and for the same analyst on a Friday afternoon.

A taxonomy is not a filing system; it is a promise to the shopper about where things will be. Eight levels of promise, kept inconsistently, is worse than three levels kept exactly.

Two unreliable signals, used against each other

The obvious build is to hand each product to a language model with the taxonomy in the prompt and ask for a path. It works impressively well in a demo and disappoints in production, for a reason worth being precise about: the model is reading semantics with no obligation to your catalogue. It will confidently return a path that reads correctly and does not exist, or pick a leaf whose name fits perfectly while its parent is in the wrong department entirely.

Fluent, plausible, unanchored.

The opposite build is to skip the model and use vector similarity alone — embed the new product, find the nearest things you have already classified, inherit their path. This is anchored to reality, which is its whole virtue. It also inherits every mistake that is already in the catalogue, and it has nothing useful to say about a product genuinely unlike anything you stock, which is precisely the case where you needed help.

Each signal fails where the other is strong, so the engine runs both. It normalises each incoming SKU into its title, description, ingredients, brand and attributes, then reads that record two ways: the model proposes a path from meaning, and similarity search grounds the proposal against products already sitting in the taxonomy.

Agreement between the two is the cheap part. Disagreement is the useful part, because disagreement is where a per-SKU confidence score comes from without anyone having to calibrate one from scratch. Structural rule checks sit alongside it and cost almost nothing: a leaf that does not exist under the proposed parent is not a judgement call, it is a broken path, and it should never reach a human queue at all.

The classifier argument, which is mostly right

There is a well-made and frequently-made case against all of this, and it goes: an LLM is the wrong tool for classification. If you have a fixed label set and labelled examples, a supervised text classifier — a linear model over character n-grams, a fastText-style embedding classifier, a gradient-boosted model over TF-IDF features — will beat a language model on cost by orders of magnitude, run in milliseconds rather than seconds, produce calibrated probabilities you can actually threshold, and give you a deterministic artefact you can version and regression-test.

I think that argument is right, and right more often than the current market admits. If you are classifying into a settled taxonomy with hundreds of examples behind every label, reaching for an LLM is an expensive way to avoid building a training set.

Where this case departs from it is on three specifics. The taxonomy had just been redesigned, so for a large share of leaves there were no labelled examples to train on — a supervised classifier needs the very thing you do not have yet. The distribution is brutally long-tailed: a grocery catalogue at this size has a small head of obvious products and thousands of leaves holding a handful of items each, which is exactly the regime where supervised models are weakest and semantic reading is strongest.

And the taxonomy is not finished, because taxonomies never are. Every structural edit invalidates a trained classifier and starts the labelling cycle again, whereas a prompt-and-similarity approach re-reads the new structure and can re-evaluate prior classifications in bulk.

The honest version of the trade-off is that we bought flexibility with money and latency. That is a fine bargain during a taxonomy redesign and a worse one two years later, once the labels have stopped moving and the corrections have piled up into the training set you originally lacked. The classifier argument is not wrong. It is early.

What ninety-five per cent buys, and what it costs

The figure the engine reaches is 95%+ accuracy across all taxonomy levels, and the catalogue it holds is 200,000+ SKUs. Put those two together and the arithmetic is unflattering: five per cent is ten thousand products, sitting somewhere they should not be, right now, in a catalogue everyone is describing as accurate.

Before anything else, be careful what that percentage is a percentage of. “Accuracy across all levels” can mean a full-path exact match, where a product only counts as correct if all eight levels are right, or it can mean an average of per-level accuracies, which is a much kinder number because levels one and two are easy and there are eight of them dividing the damage. Those are very different claims wearing the same words.

If you are evaluating anybody’s classification figure, including ours, insist on the strict reading and ask three further questions: how large the evaluated sample was, who adjudicated it, and where in the catalogue it was drawn from. That last one matters most. Sample from the fast-moving head and any engine looks excellent, because the head is the part that was never hard.

Which brings us to the decision that actually shaped the build. You cannot get to a hundred per cent, and chasing the last few points is the most expensive work available. So the useful question is not how to eliminate the ten thousand. It is where you are willing for them to be.

An error deep in the tree is usually survivable. The product sits on the shelf next to the right one, a merchandiser spots it eventually, and most shoppers never notice because search and the surrounding facets still behave. An error at the top of the tree is a different animal. Put that oat drink under Household and it is functionally not stocked. Nobody browsing for it will open that department, and the shopper’s conclusion is not “this is mis-filed” but “they do not sell it”.

That is why the confidence score routes rather than just reports. Products where the two signals agree and the structural rules pass go straight to publish. Products where they split, or where the score is low, go to a human queue — and because the score is per-SKU, the review effort concentrates on genuinely ambiguous items instead of being spread evenly across a catalogue that is mostly obvious.

Corrections then feed back: the engine learns from them and from historical patterns, so accuracy compounds as the catalogue grows.

The awkward bit is that the routing threshold is a business decision wearing the costume of a hyperparameter. Set it high and you have quietly rebuilt the manual bottleneck you were hired to remove. Set it low and errors publish themselves. There is no correct value discoverable from outside the business, because the answer depends on what a mis-filed product costs you, and that varies by category, by margin and by season. We can tell you what the score means. We cannot tell you what your tolerance is.

Minutes rather than days

The throughput change is the least subtle part of the story and the easiest to feel. New products are processed in batches the moment they land from a vendor feed, with no triage queue in front of them, so inventory reaches the site within minutes of arrival rather than sitting in a review backlog. Large vendor batches that used to be measured in months of catalogue work are measured in hours. Nothing clever there — it is what happens when you take the human out of the common case and leave them in the hard one.

The part still open

Continuous learning has a bias built into it that nobody has a clean answer for. The engine improves from corrections, and corrections come from products somebody looked at. People look at products that sell. So the head of the catalogue gets steadily more accurate while the long tail — the part with the fewest examples, the weakest signals and the highest error rate to begin with — generates almost no feedback at all.

The failure mode here is always the same shape: the measured accuracy climbs, the tail quietly does not, and the gap between the two widens the longer the system runs well. Sampling the tail deliberately is the obvious counter-move, and it costs review budget that is much easier to justify spending on products people are actually buying.

There is also the question of who owns the taxonomy now. The engine removed the analysts from the classification loop, which was the point, but it did not remove the need for judgement about the structure itself — when a new category deserves its own branch, when two leaves have collapsed into the same meaning, when eight levels is one too many for a department that never needed the depth.

Every one of those edits re-opens classifications the system already made, and the machinery to re-evaluate in bulk exists precisely because it will need to. What has not been settled is whose job it is to decide, and how often they should look.

Idea Infotech builds classification and retrieval systems for catalogues large enough that nobody can hold them in their head, and honest enough about the residual error to tell you where it lives. See the other engagements.