SCOUT: Sim-to-Real Text-Based Person Retrieval by Embedding-Space Prediction over Frozen Video Features

Abdarahmane Traoré, Andy Couturier, Éric Hervet

Embia, Computer Science Department, Université de Moncton

Code & annotations

84.25

final leaderboard mAP@10 (full system)

60.63

leaderboard mAP@10, single frozen model

~95

GPU-hours trained, all components

GPU-days for the fine-tuned CMP baseline

Overview

Text-based person retrieval asks a system to find, in a gallery of images, the one person matching a free-form caption describing their appearance, action, and scene. The sim-to-real variant in AI City Challenge 2026 Track 4 sharpens the problem: training data is diffusion-generated synthetic imagery, but the test gallery is real photographs, so a model has to generalize across that domain gap with no real labels.

The dominant approach fine-tunes a cross-encoder that jointly attends over each (caption, image) pair, powerful but expensive: the dataset's own baseline (CMP) trains for sixteen GPU-days, and a cross-encoder can't serve as a first-stage retriever since scoring an N-image gallery costs N fusion passes rather than N cached embeddings.

SCOUT asks whether a fully frozen-encoder system can compete instead. A trainable predictor reads the patch tokens of a frozen V-JEPA video encoder and maps them into the embedding space of a frozen text encoder, trained only with a bidirectional InfoNCE objective. Neither encoder is fine-tuned in the base model, so the result is a bi-encoder: gallery embeddings are computed once and queried with a dot product. Cross-attention enters only later, as an optional reranker over a short candidate list.

Architecture

A frozen self-supervised video encoder (V-JEPA 2.1 ViT-L/16) emits patch tokens. A trainable 24-layer predictor, initialized from a Qwen3.5-0.8B decoder, reads them alongside K=64 learnable prompt tokens and reads out a predicted caption embedding via a concatenated projection, one output slot per prompt (mean-pooling the prompts instead collapses at scale). Training is bidirectional InfoNCE (τ = 0.07) against a frozen text encoder's true caption embedding; that encoder is never trained.

Three findings

The right frozen text target is an alignment question, not a size question. A training-free score (mutual k-NN overlap, linear CKA, ridge-probe R²) measures how well a candidate text encoder's geometry matches the V-JEPA features, and it ranks candidates in the same order as their trained retrieval accuracy, Spearman ρ = 1.0 within the CLIP / EmbeddingGemma / PE-Core family. A 123.65M-parameter CLIP text encoder beats a larger, text-only EmbeddingGemma because its manifold is already shaped toward the image side. The rule has a falsifier: an LLM-based embedding model looks best by ridge R² yet trains into the worst retriever of the four, so the rank correlation identifies a family boundary, not a universal law.

The gap to the top teams is almost entirely rank-1 precision, not recall. The fused retriever pool already holds the ground truth in its top-128 for 99.98% of validation queries, so two precision-targeted levers, a gentle ExPLoRA adaptation of the video encoder and a training-free attribute-decomposed VLM reranker, add 2.2 points of leaderboard R@1 without touching the frozen base model's recall.

Validation gains and leaderboard gains can disagree, predictably. A local-versus-public calibration study finds that gains from adding a decorrelated signal (fusion, a new retriever, a new rerank term) grow on the harder real leaderboard, sometimes flipping sign, while gains from reordering an existing signal shrink or reverse. The per-query rank decorrelation (Spearman ρ) of a candidate addition against the existing ensemble predicts which direction a val gain will transfer.

The PAB benchmark

Track 4 evaluates on the Pedestrian Anomaly Behavior (PAB) benchmark: 1,013,605 synthetic image-caption pairs (Realistic Vision V4.0 diffusion, Qwen2-VL captions) for training, and 1,978 real-photo query captions over a 36,773-image gallery for testing, with exactly one relevant image per query. The test set is balanced one-to-one between normal and anomalous behavior, so retrieval has to couple fine-grained description matching with sensitivity to the anomaly.

Synthetic training example: a normal activity, a person dunking a basketball. — normal

Synthetic training example: an anomaly, a person lying on the grass. — anomaly

Hard pair, same person and scene: normal, jumping on a trampoline. — hard pair, normal

Hard pair, same person and scene: anomaly, falling off a trampoline. — hard pair, anomaly

A hard pair is the same generated person and scene rendered once normal and once anomalous, the caption differing only in action, used to mine harder negatives during training.

Building the recipe

Each row below adds one change to the row above, val mAP@10 on a held-out 5,000-record split. The biggest jump is the K=64 concatenated read-out (+14.3), which aggregates the stepwise growth of K from 1 to 64 plus the switch to a concat read-out; the largest single-factor gain is raising the per-GPU InfoNCE batch from 16 to 128 (+9.4). That batch is per-rank: pooling negatives across all six GPUs (127→767 negatives at identical compute) does not significantly improve on it, so the per-rank forward pass itself, not the raw negative count, is what's doing the work.

Change	Video enc.	Pred.	K / read-out	val mAP@10
CLIP-L (frozen, joint)	ViT-L/14	n/a	n/a	56.89
SigLIP2-L (frozen, joint)	ViT-L/16	n/a	n/a	68.95
base recipe	ViT-B/16	8	1 / mean	59.35
+ class balance	ViT-B/16	8	1 / mean	60.99
+ predictor depth 8→24	ViT-L/16	24	1 / mean	62.90
+ K=64, concat read-out	ViT-L/16	24	64 / concat	77.19
+ InfoNCE batch 16→128	ViT-L/16	24	64 / concat	86.57
+ pre-mined hard pairs	ViT-L/16	24	64 / concat	87.90
control: gather negs 127→767	ViT-L/16	24	64 / concat	86.79

Picking the frozen text target

Three training-free measures of X↔Y alignment, computed before any multi-GPU-hour training run, order the candidate text encoders identically to trained retrieval accuracy within the CLIP / EmbeddingGemma / PE-Core family. The falsifier (last row) is the most aligned candidate by ridge R² yet retrieves worst of the four, since InfoNCE reaches its lowest training loss on that target while generalizing worst over the gallery.

Frozen text target Y	kNN@10	CKA	ridge R²	val mAP@10
CLIP-ViT-L/14 text	0.237	0.480	0.297	77.92
EmbeddingGemma-300M	0.220	0.421	0.262	74.81
PE-Core-B-16 text	0.196	0.384	0.230	67.57
Qwen3-Emb-0.6B (falsifier)	0.218	0.444	0.308	61.37

Two precision-targeted levers

On the final leaderboard the top three teams lead SCOUT by 21–23 points at rank 1 but only 2–3 at rank 10, the signature of a precision problem rather than a recall one. A gentle ExPLoRA adaptation of the video encoder (unfreezing a few blocks, LoRA on the rest, zero-initialized so the encoder starts equal to the frozen model) and a training-free reranker that scores appearance, action, and scene separately instead of one holistic yes/no both push R@1 without touching recall.

Lever	val mAP@10	leaderboard mAP@10	leaderboard R@1
Frozen base model (CLIP-Y)	89.32	82.43	73.05
+ ExPLoRA video-encoder adaptation	92.97	82.94	73.71
+ attribute-decomposed VLM rerank	n/a	84.18	75.28
+ fusion-member swap (best submission)	n/a	84.25	75.63
Final leaderboard best (method undisclosed)	n/a	99.30	98.74

From a single model to the full system

Leaderboard progression from a single frozen model (60.63 mAP@10) through fusion, VLM reranking, and precision levers to the full system (84.18-84.25 mAP@10).

Submitted alone, the best frozen single model scores 60.63 leaderboard mAP@10. Fusing decorrelated retrievers recovers +13.6 points, three rerank signals (a VLM cross-encoder, ScoutITM, and the decomposed reranker) add a further +7.7, and retriever-pool upgrades including the ExPLoRA swap add the remaining +2.3 to reach the final 84.25.

Local validation vs. the public leaderboard

Validation mAP@10 against leaderboard mAP@10 for four intervention types, showing retention ratios from 69% to 86% and one sign flip.

A single-model retention factor of 0.70 looked stable but didn't hold across intervention types. Diversity interventions (fusion, adding a decorrelated retriever) gain more on the real leaderboard than on the near-saturated validation split, sometimes flipping sign, because a diversity gain scales with how much error remains to fix, and the real test has far more of it. Reorder-only refinements do the opposite: a rank-fusion re-blend that gained +3.0 points on a harder validation split lost 0.52 on the leaderboard.

Takeaways

Casting retrieval as prediction in embedding space over a frozen self-supervised video encoder and a frozen text encoder gives a base model that trains in under eight hours on six GPUs and reaches 89.32 val mAP@10 (92.97 with ExPLoRA). The precision levers and the fusion-and-rerank system raise a lone submission of 60.63 to 84.25 on the final leaderboard, at a fraction of a fine-tuned cross-encoder's training cost. Bigger frozen video encoders, stronger adaptation, and more fusion members all saturate on this benchmark, and replacing a calibrated signal instead of adding to it regresses. The remaining gap to the top teams is top-rank precision, and their methods are undisclosed.

The alignment criterion is validated within one encoder family on one dataset, and the calibration findings rest on one benchmark's submissions, so both are heuristics we expect to refine rather than universal laws.

Citation

@misc{traore2026scout,
  title={SCOUT: Sim-to-Real Text-Based Person Retrieval by Embedding-Space
         Prediction over Frozen Video Features},
  author={Traoré, Abdarahmane and Couturier, Andy and Hervet, Éric},
  year={2026},
  note={ECCV 2026 Workshop (AI City Challenge, Track 4), accepted},
  url={https://github.com/abtraore/SCOUT-ECCV}
}