Back
Reverse pickup the moment a rider collects a returned item from a customer's door is one of the last genuinely unverified steps in Indian e-commerce logistics. This paper documents four experiments: why image-similarity models fail at it, why a vision-language model succeeds, and what it takes to serve one on your own GPUs at national scale.
A customer requests a return. A rider arrives, performs a quality check, and marks the order picked or failed. Until recently, that quality check rested entirely on human judgement: the rider captured a few images of the product and answered a set of predefined QC questions, and the outcome followed from those answers alone.
Nothing in that loop verified item identity. There was no automated way to confirm that the rider had picked the correct product, that the QC answers reflected what was actually in front of them, or that the captured item genuinely matched the catalogue product the seller expected back.
The consequences are ordinary and expensive: seller escalations when the wrong product arrives, product-swap incidents that go undetected at the point of pickup, investigation effort after the fact, and compensation paid on items that were never correctly collected in the first place.
Riders capture a small, fixed set of images at the door: a front view, a back view, the brand tag and the MRP tag. Those images, plus a catalogue image and basic product metadata supplied by the client, are the only inputs available. Everything else lighting, framing, whether the item is folded, whether it is still inside its packaging is outside our control.
The system has to answer one question in a few seconds: does the item the rider captured match the expected catalogue product? Not a similarity score, a verdict, a confidence, and a reason a human can act on.
Catalogue images are shot to sell, not to match. A shirt listing may contain a model, a backdrop, footwear, and accessories everything except an isolated view of the shirt. Only one object in the frame is the subject; the rest actively misleads a matching model.
A top-down studio shot and a side-on doorstep capture of the same product can look like two different objects. Any method that assumes comparable geometry between the two images inherits an assumption the real world does not honour.
Motion blur, poor light, partial visibility, clutter, awkward framing. Capture quality cannot be enforced on the ground without adding time to every pickup, so the model has to absorb the variance instead.
A studio photograph of a white shirt on a model, and the same shirt folded on a floor beside a delivery carton, are the two things this system has to call identical reliably, in seconds, at national volume.
We did not arrive at one model. We arrived at four, each of which failed in a way that defined the next attempt. The sequence matters more than any single result, so we present it as a sequence.
A class of models aligns images and text in a shared embedding space, so the obvious first move was to compare a client's product description against the rider's image by similarity in that space. It worked when descriptions were specific "black polo t-shirt with white collar stripe" and failed when they were not. "Men clothing." "Item." "Product 1234." Metadata quality varies enormously between clients and within a single catalogue, and we could not make it a dependency.
Self-supervised vision transformers compare images directly, with no text involved. The pipeline was three steps: embed the catalogue image, embed the rider image, compare the two representations.
It handled shape, structure, texture, and category-level differences well: a shoe is not a shirt is not a bag. It could not reliably separate close colours, or two garments differing only in a small design detail. Those are exactly the mismatches that generate escalations on branded catalogues.
The diagnosis for the embedding weakness was catalogue noise: the representation describes the whole frame, model and background included. On our reference white-shirt case, similarity between catalogue and rider image was far too low to act on; the embedding had been dragged away from the garment by everything around it.
Open-vocabulary detection produces a box around the product from a text prompt as simple as "shirt"; a segmentation model converts that box into a pixel-accurate mask and strips everything else before embedding. We then pushed further, attempting to match corresponding regions of the two images directly.
On the same reference case, the segmented similarity barely moved. A density analysis explained why: at a loose threshold, many regions appeared to correspond, and at a stricter threshold, none survived. Inspecting the correspondences confirmed they were mostly noise rather than genuine structural alignment.
Finding: Region-level matching assumes consistent viewpoints. When part of the product is folded away, cropped, or angled out of frame, its features are not a weak signal; they are meaningless. It needs a controlled capture environment, and a doorstep is the opposite of one.
Three experiments in, the real problem surfaced. Every approach so far answered how similar these two images are. The business needs an answer to: are these the same product? And no similarity score answers that without a human to interpret it.
A vision-language model can answer the second question directly. Ours takes the product name and category, the catalogue image, and a small set of rider captures, and reasons through the comparison in natural language before committing to a verdict.
Left unconstrained, it hallucinates: it invents details it cannot see and ignores ones it can. The behaviour that mattered came from a structured, multi-step instruction that forces description before judgement, and that is as explicit about what the model should disregard as about what it should weigh. In doorstep photography, most of what sits in the frame is irrelevant, and a model that has not been told so will treat it as evidence.
The output is structured rather than free text, and includes the model's own descriptions of both images alongside its verdict. That ordering is deliberate. Because the reasoning is recorded before the conclusion, every decision carries its own audit trail an operations team can see not just that an item was rejected but what the model believed it was looking at, and disagree with it on the record.
Making the model explain itself was not a transparency nicety. It is what turned the error analysis below from guesswork into a diagnosis.
Confidence is calibrated against defined bands rather than left free-form, so that a partially obscured but probably-matching item and a clearly identical one do not return the same number. That calibration is what makes tiered downstream actions possible at all.
The four experiments did not converge on a winner. They converged on the observation that the right approach depends on the catalogue, not on the leaderboard.
Before any traffic reached it, the final prompt iteration was scored against a manually labelled set of roughly 700 pickups drawn from a branded catalogue, with human ground truth on every case. That benchmark is maintained internally and rerun on every prompt change; its purpose is to make design decisions measurable rather than to produce a headline number.
Performance is not uniform across product types. A small number of categories are structurally harder for reasons that have nothing to do with the model: some product families are near-identical across SKUs at doorstep image quality, and some are conventionally photographed at angles riders do not reproduce. Those categories are tracked separately, both to get a cleaner read on where the reasoning is genuinely working and because they are the clearest candidates for a specialised path.
The failure analysis proved more useful than any accuracy figure. Two error classes, each with a dominant pattern, and each pointing at a different kind of fix.
The largest single bucket is not a perception failure at all. The model describes both images accurately and then lets matching surface attributes ("both blue", "both checked") outweigh the difference it recorded itself one line earlier. Behind that sit genuinely marginal cases: small design variations, close colours, subtle differences in print, items photographed folded, and items still inside packaging where the product is barely visible.
The single most actionable finding: In the largest failure bucket, the perception is right, and the aggregation is wrong. That is a decision-rule problem rather than a model-capacity problem, which means it is fixable without retraining the cheapest class of fix available in any vision system.
Here the pattern inverts. The leading cause is the model itself oversensitivity, or a detail asserted that the image does not support. But a substantial share of the remainder traces back not to the model but to the photograph: captures taken too close, items folded, packaging in the way, motion blur.
That split matters operationally. A meaningful portion of false rejects originates at the capture step, which is the cheapest place in the entire system to spend engineering effort and the only place where an improvement helps every downstream approach at once.
Both classes point the same way. The most valuable improvements available to us live in the decision rules and in the capture workflow rather than in the model architecture, which is fortunate, because those are also the two cheapest things to change.
Every prompt revision was scored against the same labelled benchmark, which turned prompt engineering from a matter of taste into a matter of measurement. That single practice mattered more than any individual wording choice: without a fixed evaluation set, prompt work becomes an exchange of opinions about outputs nobody has counted.
One tension dominated and never went away. Making the model stricter catches more wrong pickups and rejects more good ones; relaxing it does the reverse. No phrasing escapes that trade-off. Where you choose to sit on it is a business decision about which of the two errors costs more, and for a reverse pickup, the two costs are genuinely different in kind, not just in size.
A fixed, human-labelled evaluation set is the prerequisite. Intuitions about what a prompt change will do to model behaviour were wrong often enough that we stopped trusting them without a score attached.
Both prompt detail and image count have a point of diminishing return, past which additional material competes for the model's attention rather than adding to it. The optimum is found empirically, not assumed.
Input resolution improves fine-detail discrimination and costs inference time directly. The operating point is fixed by the latency a doorstep interaction can absorb, which is set by operations rather than by engineering.
At tens of thousands of validations a day, the deployment question stops being incidental. We self-host the model rather than call an external API for data residency, for unit economics at volume, and because latency behaviour under load becomes something we control rather than something we discover.
Three choices carried most of the weight, and they are worth stating as principles rather than as a parts list.
The model is quantised so that it fits comfortably on a single GPU. The instinctive reading of that is "the model got smaller," which misses the point entirely. Model weights and the inference cache share the same GPU memory, and that cache is the working memory for every request in flight. Vision requests are unusually expensive there, because an image expands into far more context than a comparable amount of text.
Compression does not shrink the cache. It makes room for more of it, and cache headroom is what concurrency actually is. The right way to size a deployment is therefore backwards from how many simultaneous image-bearing requests you need to serve, not forwards from how large the model file is.
We began on a runtime that is excellent for local experimentation and wrong for production: it processes requests one at a time and allocates memory in fixed blocks with real waste. The production engine batches requests in parallel and allocates memory in small blocks on demand, the same idea an operating system uses for virtual memory. Moving between them changed throughput by an order of magnitude on identical hardware and an identical model. The experimentation runtime also proved unstable under sustained load; the production one did not.
Its second contribution is continuous batching. As soon as one request finishes, its slot is filled immediately rather than waiting for the whole batch to drain. The GPU is never idle, which raises throughput and lowers average latency at the same time- the rare optimisation that does not trade one against the other.
The last choice is the least glamorous, and the one most often skipped. The model runs as a containerised service under standard orchestration, with the same health checks, autoscaling, self-healing and dashboards as any other production system. Latency, throughput and cache pressure are monitored continuously, because a model that quietly degrades under load is indistinguishable from one that is working until someone looks.
The system has been running PAN India across both routes for months, at a scale of tens of thousands of validations per day, with daily monitoring of outcome rates and of the flag rate that surfaces potential mismatches while the rider is still at the door.
Two things are worth reporting from live operation. First, stability: the flag rate holds steady in a narrow band week over week, with no drift or degradation at full volume. A model that quietly gets noisier under production traffic is a model that will eventually be switched off, and this one has not. Second, impact:
Loss accepted per order on reverse pickups fell substantially in the first week after go-live, measured against the week before at constant order volume.
Restricted to the design, pattern and colour mismatches the system is actually built to catch, the reduction is substantially larger.
Unchanged post go-live, confirming the reduction came from catching mismatches at the door rather than from displacing the problem downstream.
There is also a measurable second layer of value still on the table. In a share of cases, the system flags a likely mismatch and the item is collected regardless. Converting that behaviour through better in-flow prompting and confidence-tiered actions rather than a single binary flag represents a further meaningful increment on what has already been realised, without any change to the model itself.
Reverse pickup validation turned out not to be an image-similarity problem at all. The obstacles were noisy catalogue data, unstructured captures, inconsistent viewpoints, and a business question: is this the same product? That no distance metric answers on its own.
Getting there took four full iterations. Text–image matching exposed a metadata dependency we could not rely on. Vision embeddings solved category and failed design. Detection and segmentation proved that isolating the product helps and that region-level matching is too fragile for uncontrolled conditions. A vision-language model finally reframed the task as reasoning rather than measurement.
None of them fully replaced the one before. Production today runs segmentation-plus-embeddings on one route and vision-language reasoning on the other, because the right answer depended on the catalogue rather than on picking a universal best model. And the self-hosted infrastructure quantisation, continuous batching, and the discipline of tuning a prompt against a real trade-off between catching mismatches and rejecting good ones is what turned a research result into something that runs, every day, at the doorstep.
Hash Tags :
#shadowfax #reversepickups #whitepaper #reverselogistics #returnpickups #returnparcel #visualaireversepickups #visionlanguage #qualitycontrol #AIreversepickup
Related Blogs
