Multi-size pairs: the real-world ground truth VTO still doesn't have

Virtual try-on has a strange failure mode. Ask a state-of-the-art model to render the same person in a size 30 and a size 32 of the same jeans, and you will very often get two nearly identical images — both looking like a well-fitted product photo. The model isn't broken. It is doing exactly what its training data taught it to do.

The field has started to name this problem. The recent FIT dataset from UW and Google Research (SIGGRAPH 2026) puts it plainly: current VTO methods "default to generating well-fitted results regardless of the garment or person size" — and it takes what its authors describe as the first steps toward solving this, with over 1M synthetic triplets carrying precise body and garment measurements. We think FIT is exactly right about the problem, and that its synthetic strategy defines, by construction, the natural next step: real-world multi-size ground truth, on real bodies, in real garments. That is what we are building — starting with denim, a category outside FIT's upper-garment scope, and the one where body–garment interaction is hardest to simulate.

In this post we describe the data structure at the center of this — same-person, same-garment, adjacent-size captures under controlled conditions — show what it looks like in practice, and propose an evaluation for size fidelity on real-world data.

1. The causal hole in VTO training data

The standard training unit for try-on models is a pair: a flat garment image and a photo of a person wearing that garment. Datasets like VITON-HD and DressCode assembled tens of thousands of these pairs, largely from e-commerce photography, and they made the current generation of try-on models possible.

But look at what is not observed in that pair:

• ●The person's body measurements. The model sees a silhouette, not a waist circumference.
• ●The garment's size. The catalog image is size-agnostic; the worn image is one unlabeled size on one unlabeled body.
• ●The fit outcome. Whether the garment fits this person — and where it strains or gaps — is invisible.

The training pair says "this garment, on a person, looks like this." It never says why it looks like that. Size, body, and fabric — the three variables that jointly determine drape — are all unobserved. A model trained on this data learns the statistics of how worn clothing looks, and because e-commerce photography overwhelmingly shows well-fitted garments on sample-size models, that statistic has a single dominant mode: clothes that fit.

The consequence is a form of mode collapse along the size axis. Condition the model on "size S" versus "size L" and the output distribution barely moves, because nothing in training ever forced the size variable to matter. The model was never shown two images whose only difference was size.

Here is the landscape as of mid-2026:

Two things stand out. First, the field now agrees that size supervision matters — FIT's existence is the proof. Second, the size-supervised data that exists at scale is synthetic, while the real-world datasets with size variation (SIZER, SV-VTO) are three orders of magnitude smaller and were captured in lab settings years ago. Real-world multi-size ground truth at training scale — the bottom-right region of this table — remains empty. That is the region we operate in.

2. Multi-size pairs are controlled experiments, not just data

The fix is conceptually simple: hold everything constant and vary only size.

A multi-size pair (sometimes a triplet) is captured as follows: the same contributor, under the same lighting, with the same camera position, wearing the same SKU in adjacent sizes, minutes apart, in a standardized pose set. The result is a set of images whose pixel-level differences are attributable to exactly one cause: the size change.

This is worth stating in experimental-design language, because that is what it is. Every uncontrolled try-on photo on the internet is an observational sample — size effects are hopelessly confounded with body, pose, lighting, and camera. A multi-size capture is an interventional sample: we intervene on one variable and observe the outcome. For learning a causal relationship (size → drape), a modest number of interventional samples is worth a very large number of observational ones.

Figure 1 — hero figure

Two-panel image: the same contributor wearing sizes 23 and 24 of the same jeans SKU (Levi's 501 Original Fit, 100% cotton), front view, same fitting session, panels aligned via background homography and anatomical anchoring. Annotations: (a) button-fly closure failure on the size 23 — hip delta −1.25″ on rigid denim; (b) the 23's waistband reaches the same seating height as the 24 — but only because the fly is open, with the center-front sagging at the gap; (c) fabric tension across hip and thigh, relaxed in the 24, which closes and seats at the waist.

CaptionEverything here is constant except the size label. The nominal waist fits in both sizes — but the hip is the binding constraint, and with zero elastane there is no stretch to absorb it: the only way the 23 reaches full height is by not closing. This is the boundary every fit model needs to learn, captured as it actually happens.

Why adjacent sizes

We deliberately capture adjacent sizes (S/M, or a 23/24) rather than distant ones (S/L). This is a data-efficiency argument. The differences between adjacent sizes are small — an inch or less at each measurement point — and that is precisely why they are informative. Fit decisions happen at boundaries: nobody agonizes between an S and an XXL; they agonize between the two sizes that both almost work. A model that must distinguish adjacent sizes is forced to learn the fine-grained visual signals of fit — strain wrinkle onset, waistband gap, rise tension, whether a button fly closes at all — rather than the coarse silhouette change that any model can capture. Adjacent-size pairs concentrate supervision exactly where the decision boundary lives. Figure 1 is a live example: one size up or down from the boundary and the session would have taught us far less.

Real capture vs. simulation — after FIT

The obvious objection: simulation can generate unlimited multi-size data, and FIT has now done so at 1M scale with an impressive pipeline (programmatic garment generation, physics-based draping, re-texturing to photorealism). We consider synthetic and real data complementary, and FIT is the strongest version of the synthetic strategy to date. But three gaps are structural, not incidental — and they are largest precisely in denim.

First, garment fidelity: simulation needs the garment's pattern and measured fabric mechanics as input. For real-world denim, with its brand-specific patterns, washes, and elastane blends, those inputs are simply unavailable to any simulator.

Second, body–garment interaction: the hardest part of denim fit is soft tissue — waistband compression, how stretch denim loads and recovers over the hip, what happens when a rigid rise meets a real seated posture. Simulators drape cloth over bodies that are, mechanically, near-rigid mannequins. The sim-to-real residual is largest exactly where jeans-fit decisions are made — which may be why fit-aware pipelines have sensibly started with upper garments, where this interaction is mildest, and why bottoms are where real capture earns its cost.

Third, and most simply: synthetic data cannot audit itself. Whatever fraction of real drape behavior a simulation pipeline captures, measuring that fraction requires real multi-size ground truth. Real captures are not just an alternative training source — they are the only possible evaluation set for the synthetic ones.

3. What this data unlocks

We see three immediate uses, in increasing order of novelty.

3.1 Size-conditional generation that actually conditions

With multi-size pairs, a generative model can be trained (or fine-tuned) with explicit supervision that the size token must change the output — and change it in the observed way. The training signal is no longer "produce a plausible worn image" but "produce the M image given the S image and the size delta," with a real M image as the target. Contrastive setups fall out naturally: the S and M captures of the same session are a positive pair for identity and a negative pair for fit state.

3.2 Fit classification with clean labels

Because each session captures the same body in multiple sizes, the dataset directly encodes comparative fit ground truth. Combined with per-session fit annotations (where the garment strained, whether the waistband closed comfortably), this trains fit-assessment models on exactly the question consumers ask: not "does this look nice" but "which of these two sizes is right for this body."

3.3 The benchmark that closes the loop: real-world size fidelity

Perhaps the most immediate use is evaluation. FIT ships a benchmark for fit-aware generation — on synthetic ground truth. The open question it leaves is the one that decides deployment: does a fit-aware model, however trained, respect size on real bodies in real garments? Real multi-size captures make the following metric possible:

This metric is agnostic about training data — models trained on synthetic multi-size data (the FIT recipe), pseudo-pairs, or anything else can all be scored on it. That is the point: it measures whether the sim-to-real gap in fit has actually been crossed. We intend to publish baseline numbers as our dataset grows, and we would welcome others evaluating against it.

Figure 2

Three panels: the aligned size-23 and size-24 captures, and their garment-region segmentation difference (background-feature homography → anatomical anchoring → per-row stance normalization, so pose and camera variation between shots is factored out). Differences concentrate at the thigh (the 24 runs wider, matching its +0.2″ ease against the 23's −0.4″), at the hem (the 23's fabric stacks at the ankle), and at the open fly, where the 23's closure failure leaves a fabric gap.

CaptionReal size deltas are structured and localized — and they land exactly where the measurement deltas in Section 4 predict. Whether generated try-on images preserve these deltas is, we argue, the evaluation the field is missing (Section 3.3).

4. The metadata layer: from images to body–garment deltas

Images alone are half of the ground truth. Each capture session in our dataset carries a structured metadata record:

Body side (collected at onboarding, verified against capture): waist, hip, thigh, inseam, height — the measurement set that determines denim fit.

Garment side (verified SKU, not self-reported): brand, model, size, and the flat measurements for that size; fiber composition including elastane percentage, which for denim changes fit behavior more than a full size step does.

The join of these two records yields what we consider the dataset's most useful derived quantity: the body–garment delta — the signed ease (garment measurement minus body measurement) at each anatomical point, per size. Here is the record behind Figure 1:

From the actual capture session shown in Figure 1 (Levi's 501 Original Fit, 100% cotton; garment measurements taped on the physical garments; contributor consented to publication). Read the waist row alone and both sizes fit. The hip row is the real story: −1.25″ of negative ease on rigid denim with zero elastane is why the size 23's button fly will not close — the nominal size is indexed to the waist, but the hip is the binding constraint. Note also that the "size 23" measures 24.25″ at the waist: a 1.25″ gap between nominal and actual, which is itself a data point on why measured garments, not size labels, are the usable supervision.

This is the supervision structure fit modeling has been missing: for every image, the numerical cause (the delta) and the visual effect (the drape, the strain, the gap) — and for every body, that relationship observed at two or more points along the size axis.

5. What we are building

Fittings Labs is building this dataset at scale, with an initial focus on denim — the category where fit failure is most common, most expensive, and, conveniently, where the fabric's rich texture makes high-quality geometric normalization of real-world captures most tractable. Every session is consent-backed and captured through a guided mobile flow; our normalization pipeline (frame extraction, camera-axis alignment, cross-session body anchoring) is the subject of an upcoming post.

If you are training or evaluating try-on and fit models and want to see what controlled multi-size ground truth looks like, we share sample batches with research and enterprise teams — reach us at fittingslabs.co.

Appendix: capture protocol summary — same contributor, same SKU in adjacent sizes, same session; standardized pose set (front / side / back); verified SKU via tag capture; body measurements at onboarding; per-session fit annotation. Full protocol details available under NDA.