SmolRGPT: Efficient Spatial Reasoning for Warehouse Environments with 600M Parameters

A warehouse robot that can act on a scene also needs to talk about it: how many pallets sit in the leftmost buffer area, how far one pallet is from another, which side of a transporter an object falls on, and which pallet an empty transporter should pick up next. SmolRGPT is a multimodal system built to answer exactly these spatial questions from RGB and depth images of real warehouse environments.

The system pairs a frozen visual feature extractor (SigLIP2) with trainable connector and refiner modules that process RGB and depth features separately. Region masks select the objects a question refers to, and mask pooling aggregates their features into compact region tokens. Those tokens, together with the question, are fed to a small vision-language model (SmoILM2) that integrates visual and textual information and produces a natural-language answer grounded in the referenced regions.

The bet is efficiency. At 600M parameters the whole system is one to three orders of magnitude smaller than the GPT-4-class and 7B–34B models it benchmarks against, yet it took 3rd place in the AI City Challenge 2025 Warehouse Spatial Intelligence track and beats every model under 7B on SpatialRGPT-Bench's qualitative tasks, a size that makes on-premise, resource-constrained deployment realistic.

SmolRGPT placed 3rd of 9 teams in AI City Challenge 2025 Track 3 with a final S1 score of 90.68, within 1.3 points of 2nd place. The per-task breakdown shows where the small model holds up: 99.8% accuracy on left–right direction tasks, 92.76% on counting (RMSE 0.0750) thanks to the depth refiner, 88.02% on multiple-choice spatial reasoning, and 82.13% on distance estimation (RMSE 0.4740), the hardest task and the one where RGB+depth fusion matters most.

To test spatial reasoning beyond the warehouse, we trained SmolRGPT on a mix of the OSD and Warehouse datasets and evaluated zero-shot-style on SpatialRGPT-Bench. With 600M parameters it reaches a 65.6% qualitative average, ahead of GPT-4 (57.8%), GPT-4V (58.1%), and LLaVA-v1.6-34B (43.9%), and strongest on behind/front (79.0%) and tall/short (74.1%). The 7B SpatialRGPT models score higher overall but at more than ten times the parameter count.

On the quantitative tasks, SmolRGPT matches or beats the larger models on direct distance (35.8 vs. GPT-4V's 29.7) and direction, while width and height estimation remain the weak spots, the same pattern as on Track 3, where distance was the hardest task class.

Four task types from the warehouse benchmark (counting, distance estimation, left–right relations, and multiple-choice grounding), each shown with the masked frame, the question, SmolRGPT's prediction, and the ground truth.

A 600M-parameter system can sit at the top of a warehouse spatial-intelligence leaderboard: 3rd of 9 teams in AIC 2025 Track 3 at 90.68, and ahead of every sub-7B model, as well as GPT-4 and GPT-4V, on SpatialRGPT-Bench's qualitative tasks. The remaining gap to the 7B SpatialRGPT models is concentrated in the fine-grained quantitative estimates.

Depth is what makes the quantitative tasks work. Separate RGB and depth refiners over a shared frozen SigLIP2 backbone drive the 92.76% counting accuracy (RMSE 0.0750) and the 99.8% left–right accuracy on Track 3, while distance estimation (82.13% on Track 3, and width/height on SpatialRGPT-Bench) remains the open problem.

Because only the connectors, refiners, and the small decoder train, the whole pipeline stays cheap enough for on-premise, resource-constrained deployment: the point of building a warehouse reasoner at 600M in the first place.