Back
Reverse pickup, the moment a rider collects a returned item from a customer's door, is one of the last genuinely unverified steps in Indian e-commerce logistics. This post documents four experiments: why image-similarity models fail at it, and why a vision-language model finally succeeds.
A customer requests a return. A rider arrives, performs a quality check, and marks the order as picked or failed. Until recently, that quality check rested entirely on human judgement: the rider captured a few images of the product and answered a set of predefined QC questions, and the outcome followed from those answers alone.
Nothing in that loop verified item identity. There was no automated way to confirm that the rider had picked the correct product, that the QC answers reflected what was actually in front of them, or that the captured item genuinely matched the catalogue product the seller expected back.
The consequences are ordinary and expensive: seller escalations when the wrong product arrives, product-swap incidents that go undetected at the point of pickup, investigation effort after the fact, and compensation paid on items that were never correctly collected in the first place.
Riders capture a small, fixed set of images at the door: a front view, a back view, the brand tag, and the MRP tag. Those images, plus a catalogue image and basic product metadata supplied by the client, are the only inputs available. Everything else lighting, framing, whether the item is folded, whether it is still inside its packaging is outside our control.
The system has to answer one question in a few seconds: does the item the rider captured match the expected catalogue product? Not a similarity score, but a verdict, a confidence, and a reason a human can act on.
A. Catalogue noise. Catalogue images are shot to sell, not to match. A shirt listing may contain a model, a backdrop, footwear, and accessories everything except an isolated view of the shirt. Only one object in the frame is the subject; the rest actively misleads a matching model.
B. Viewpoint mismatch. A top-down studio shot and a side-on doorstep capture of the same product can look like two different objects. Any method that assumes comparable geometry between the two images inherits an assumption the real world does not honour.
C. Capture conditions. Motion blur, poor light, partial visibility, clutter, awkward framing. Capture quality cannot be enforced on the ground without adding time to every pickup, so the model has to absorb the variance instead.
A studio photograph of a white shirt on a model, and the same shirt folded on a floor beside a delivery carton, are the two things this system has to call identical reliably, in seconds, at national volume.
We did not arrive at one model. We arrived at four, each of which failed in a way that defined the next attempt. The sequence matters more than any single result, so we present it as a sequence.
A class of models aligns images and text in a shared embedding space, so the obvious first move was to compare a client's product description against the rider's image by similarity in that space. It worked when descriptions were specific, "black polo t-shirt with white collar stripe," and failed when they were not. "Men's clothing." "Item." "Product 1234." Metadata quality varies enormously between clients and within a single catalogue, and we could not make it a dependency.
Self-supervised vision transformers compare images directly, with no text involved. The pipeline was three steps: embed the catalogue image, embed the rider image, compare the two representations.
It handled shape, structure, texture, and category-level differences well: a shoe is not a shirt is not a bag. It could not reliably separate close colours or two garments differing only in a small design detail. Those are exactly the mismatches that generate escalations on branded catalogues.
The diagnosis for the embedding weakness was catalogue noise: the representation describes the whole frame, model and background included. On our reference white-shirt case, similarity between catalogue and rider image was far too low to act on; the embedding had been dragged away from the garment by everything around it.
Open-vocabulary detection produces a box around the product from a text prompt as simple as "shirt"; a segmentation model converts that box into a pixel-accurate mask and strips everything else before embedding. We then pushed further, attempting to match corresponding regions of the two images directly.
On the same reference case, the segmented similarity barely moved. A density analysis explained why: at a loose threshold, many regions appeared to correspond, and at a stricter threshold, none survived. Inspecting the correspondences confirmed they were mostly noise rather than genuine structural alignment.
Finding: Region-level matching assumes consistent viewpoints. When part of the product is folded away, cropped, or angled out of frame, its features are not a weak signal; they are meaningless. It needs a controlled capture environment, and a doorstep is the opposite of one.
Three experiments in, the real problem surfaced. Every approach so far answered how similar these two images are. The business needs an answer to: are these the same product? And no similarity score answers that without a human to interpret it.
A vision-language model can answer the second question directly. Ours takes the product name and category, the catalogue image, and a small set of rider captures, and reasons through the comparison in natural language before committing to a verdict.
Left unconstrained, it hallucinates: it invents details it cannot see and ignores ones it can. The behaviour that mattered came from a structured, multi-step instruction that forces description before judgement, and that is as explicit about what the model should disregard as about what it should weigh. In doorstep photography, most of what sits in the frame is irrelevant, and a model that has not been told so will treat it as evidence.
The output is structured rather than free text, and includes the model's own descriptions of both images alongside its verdict. That ordering is deliberate. Because the reasoning is recorded before the conclusion, every decision carries its own audit trail; an operations team can see not just that an item was rejected, but what the model believed it was looking at, and disagree with it on the record.
Making the model explain itself was not a transparency nicety. It is what turned the error analysis below from guesswork into a diagnosis.
Confidence is calibrated against defined bands rather than left free-form, so that a partially obscured but probably-matching item and a clearly identical one do not return the same number. That calibration is what makes tiered downstream actions possible at all.
The four experiments did not converge on a winner. They converged on the observation that the right approach depends on the catalogue, not on the leaderboard.
Catalogues built largely from unbranded, visually distinct inventory tolerate the coarser embedding approach and reward it with lower latency and materially lower cost per validation at high volume. Branded catalogues are the opposite case: near-identical SKUs, subtle design variation, and escalations concentrated in exactly the discriminations embeddings cannot make. Those route to the vision-language model and absorb the higher inference cost, because that is where the cost lives anyway.
Before any traffic reached it, the final prompt iteration was scored against a manually labelled set of 50,000+ images drawn from a branded catalogue, with human ground truth on every case. That benchmark is maintained internally and rerun on every prompt change; its purpose is to make design decisions measurable rather than to produce a headline number.
Performance is not uniform across product types. A small number of categories are structurally harder for reasons that have nothing to do with the model: some product families are near-identical across SKUs at doorstep image quality, and some are conventionally photographed at angles riders do not reproduce. Those categories are tracked separately, both to get a cleaner read on where the reasoning is genuinely working and because they are the clearest candidates for a specialised path.
The failure analysis proved more useful than any accuracy figure. Two error classes, each with a dominant pattern, each pointing at a different kind of fix.
Wrong pickups the model approved. The largest single bucket is not a perception failure at all. The model describes both images accurately and then lets matching surface attributes ("both blue", "both checked") outweigh the difference it recorded itself one line earlier. Behind that sit genuinely marginal cases: small design variations, close colours, subtle differences in print, items photographed folded, and items still inside packaging where the product is barely visible.
The single most actionable finding: in the largest failure bucket, the perception is right, and the aggregation is wrong. That is a decision-rule problem rather than a model-capacity problem, fixable without retraining, the cheapest class of fix available in any vision system.
Correct pickups the model rejected. Here the pattern inverts. The leading cause is the model itself: oversensitivity, or a detail asserted that the image does not support. But a substantial share of the remainder traces back not to the model but to the photograph: captures taken too close, items folded, packaging in the way, motion blur.
That split matters operationally. A meaningful portion of false rejects originates at the capture step, which is the cheapest place in the entire system to spend engineering effort and the only place where an improvement helps every downstream approach at once.
Both classes point the same way. The most valuable improvements available to us live in the decision rules and in the capture workflow rather than in the model architecture, fortunately, because those are also the two cheapest things to change.
Every prompt revision was scored against the same labelled benchmark, which turned prompt engineering from a matter of taste into a matter of measurement. That single practice mattered more than any individual wording choice: without a fixed evaluation set, prompt work becomes an exchange of opinions about outputs nobody has counted.
One tension dominated and never went away. Making the model stricter catches more wrong pickups and rejects more good ones; relaxing it does the reverse. No phrasing escapes that trade-off. Where you choose to sit on it is a business decision about which of the two errors costs more, and for reverse logistics, the two costs are genuinely different in kind, not just in size.
1. Measure, don't reason. A fixed, human-labelled evaluation set is the prerequisite. Intuitions about what a prompt change will do to model behaviour were wrong often enough that we stopped trusting them without a score attached.
2. More input is not better input. Both prompt detail and image count have a point of diminishing return, past which additional material competes for the model's attention rather than adding to it. The optimum is found empirically, not assumed.
3. Every quality dial costs latency. Input resolution improves fine-detail discrimination and costs inference time directly. The operating point is fixed by the latency a doorstep interaction can absorb, which is set by operations rather than by engineering.
Reverse pickup validation turned out not to be an image-similarity problem at all. The obstacles were noisy catalogue data, unstructured captures, inconsistent viewpoints, and a business question: is this the same product? That no distance metric answers on its own.
Getting there took four full iterations. Text–image matching exposed a metadata dependency we could not rely on. Vision embeddings solved category and failed design. Detection and segmentation proved that isolating the product helps and that region-level matching is too fragile for uncontrolled conditions. A vision-language model finally reframed the task as reasoning rather than measurement.
None of them fully replaced the one before. Production today runs segmentation-plus-embeddings on one route and vision-language reasoning on the other, because the right answer depended on the catalogue rather than on picking a universal best model.
In the next post, we cover how we serve this model in production at national scale: quantisation, batching, and treating a vision-language model as ordinary infrastructure.
Hash Tags :
#shadowfax #reversepickup #visionaimodel
Related Blogs
