SCOUT: Sim-to-Real Text-Based Person Retrieval by Embedding-Space Prediction over Frozen Video Features

Overview

Text-based person retrieval asks a system to find, in a gallery of images, the one person matching a free-form caption describing their appearance, action, and scene. The sim-to-real variant in AI City Challenge 2026 Track 4 sharpens the problem: training data is diffusion-generated synthetic imagery, but the test gallery is real photographs, so a model has to generalize across that domain gap with no real labels.

The dominant approach fine-tunes a cross-encoder that jointly attends over each (caption, image) pair, powerful but expensive: the dataset's own baseline (CMP) trains for sixteen GPU-days, and a cross-encoder can't serve as a first-stage retriever since scoring an N-image gallery costs N fusion passes rather than N cached embeddings.

SCOUT asks whether a fully frozen-encoder system can compete instead. A trainable predictor reads the patch tokens of a frozen V-JEPA video encoder and maps them into the embedding space of a frozen text encoder, trained only with a bidirectional InfoNCE objective. Neither encoder is fine-tuned in the base model, so the result is a bi-encoder: gallery embeddings are computed once and queried with a dot product. Cross-attention enters only later, as an optional reranker over a short candidate list.

Architecture

Three findings

1. The right frozen text target is an alignment question, not a size question. A training-free score (mutual k-NN overlap, linear CKA, ridge-probe R²) measures how well a candidate text encoder's geometry matches the V-JEPA features, and it ranks candidates in the same order as their trained retrieval accuracy, Spearman ρ = 1.0 within the CLIP / EmbeddingGemma / PE-Core family. A 123.65M-parameter CLIP text encoder beats a larger, text-only EmbeddingGemma because its manifold is already shaped toward the image side. The rule has a falsifier: an LLM-based embedding model looks best by ridge R² yet trains into the worst retriever of the four, so the rank correlation identifies a family boundary, not a universal law.¹
2. The gap to the top teams is almost entirely rank-1 precision, not recall. The fused retriever pool already holds the ground truth in its top-128 for 99.98% of validation queries, so two precision-targeted levers, a gentle ExPLoRA adaptation of the video encoder and a training-free attribute-decomposed VLM reranker, add 2.2 points of leaderboard R@1 without touching the frozen base model's recall.
3. Validation gains and leaderboard gains can disagree, predictably. A local-versus-public calibration study finds that gains from adding a decorrelated signal (fusion, a new retriever, a new rerank term) grow on the harder real leaderboard, sometimes flipping sign, while gains from reordering an existing signal shrink or reverse. The per-query rank decorrelation (Spearman ρ) of a candidate addition against the existing ensemble predicts which direction a val gain will transfer.²

The PAB benchmark

Track 4 evaluates on the Pedestrian Anomaly Behavior (PAB) benchmark: 1,013,605 synthetic image-caption pairs (Realistic Vision V4.0 diffusion, Qwen2-VL captions) for training, and 1,978 real-photo query captions over a 36,773-image gallery for testing, with exactly one relevant image per query. The test set is balanced one-to-one between normal and anomalous behavior, so retrieval has to couple fine-grained description matching with sensitivity to the anomaly.

Building the recipe

Each row below adds one change to the row above, val mAP@10 on a held-out 5,000-record split. The biggest jump is the K=64 concatenated read-out (+14.3), which aggregates the stepwise growth of K from 1 to 64 plus the switch to a concat read-out; the largest single-factor gain is raising the per-GPU InfoNCE batch from 16 to 128 (+9.4). That batch is per-rank: pooling negatives across all six GPUs (127→767 negatives at identical compute) does not significantly improve on it, so the per-rank forward pass itself, not the raw negative count, is what's doing the work.

Picking the frozen text target

Three training-free measures of X↔Y alignment, computed before any multi-GPU-hour training run, order the candidate text encoders identically to trained retrieval accuracy within the CLIP / EmbeddingGemma / PE-Core family. The falsifier (last row) is the most aligned candidate by ridge R² yet retrieves worst of the four, since InfoNCE reaches its lowest training loss on that target while generalizing worst over the gallery.

Two precision-targeted levers

On the final leaderboard the top three teams lead SCOUT by 21–23 points at rank 1 but only 2–3 at rank 10, the signature of a precision problem rather than a recall one. A gentle ExPLoRA adaptation of the video encoder (unfreezing a few blocks, LoRA on the rest, zero-initialized so the encoder starts equal to the frozen model) and a training-free reranker that scores appearance, action, and scene separately instead of one holistic yes/no both push R@1 without touching recall.

From a single model to the full system

Local validation vs. the public leaderboard

Takeaways

Casting retrieval as prediction in embedding space over a frozen self-supervised video encoder and a frozen text encoder gives a base model that trains in under eight hours on six GPUs and reaches 89.32 val mAP@10 (92.97 with ExPLoRA). The precision levers and the fusion-and-rerank system raise a lone submission of 60.63 to 84.25 on the final leaderboard, at a fraction of a fine-tuned cross-encoder's training cost. Bigger frozen video encoders, stronger adaptation, and more fusion members all saturate on this benchmark, and replacing a calibrated signal instead of adding to it regresses. The remaining gap to the top teams is top-rank precision, and their methods are undisclosed.