Feature Puzzles

Single-Image Supervision for Multi-View 3D-Aware Segmentation

Jiahao Ma1 Qiang Zhang2 Miaomiao Liu1 David Ahmedt-Aristizabal3 Chuong Nguyen3

1Australian National University 2University of Science and Technology of China 3CSIRO, Australia

Paper BibTeX Code

Video

Abstract

Humans can easily recognize the same objects across different viewpoints of a 3D scene, whereas teaching models such multi-view-consistent perception typically requires videos or calibrated multi-view captures, which are considerably more expensive and less diverse than large-scale monocular data. In this paper, we introduce Feature Puzzles, a novel framework for learning multi-view-consistent segmentation using only single-image semantic segmentation supervision. Specifically, we propose a simple yet effective pipeline that constructs multi-view-consistent segmentation pseudo-labels through image cropping and geometric transformations. We further introduce a lightweight, plug-and-play network that leverages the multi-view-consistent image features learned by existing pretrained 3D foundation models and the generated pseudo-labels to perform multi-view-consistent segmentation. Importantly, our flexible framework supports various types of prompts provided from one or multiple views and can be readily extended to open-vocabulary 3D segmentation. Under this single-image supervision, our feed-forward model achieves the best mIoU on NVOS, remains competitive on SPIn-NeRF and ScanNet++ against baselines using per-scene optimization or in-domain multi-view training, and demonstrates zero-shot transfer to open-vocabulary 3D segmentation.

  • Multi-view supervision from one image. Overlapping crops of a single image become synthetic views. Each is tied to the parent by a known transform, so their labels agree on every overlap by construction, with no video and no multi-view annotation.
  • Plug-and-play on a 3D foundation model. Lightweight decoders sit on a pretrained VGGT-Omega backbone. Its low-rank adapters switch off at inference, so the original depth and camera outputs are kept.
  • One model, many prompts. Mask, point or box prompts from one or more views return masks on every view in a single forward pass, and the same features extend to open-vocabulary 3D segmentation.

Method

1

Feature Puzzles: multi-view supervision from a single image

A parent image is cut into overlapping crops, and every crop is treated as a view. Labels are computed once on the parent and pulled back into each view through the same map that made the crop.

Feature Puzzles: one parent image, cropped into views whose masks and features are pulled back from the parent FEATURE PUZZLES · ONE IMAGE, MANY CONSISTENT VIEWS Parent image one RGB(-D) image, crops sampled on it labelled once, on the parent SAM masks DINO features Crop & warp known map πn no pose estimation optional: depth + 6-DoF 3D augmentation Pull back ℓ ∘ πn into every view VIEWS x1 … xN MASKS · instance m is the same object in every view FEATURES · DINO targets from the parent ℓi(πi−1(p)) = ℓj(πj−1(p)) = ℓ(p) · every overlap agrees by construction

Feature Puzzles. Crops of one parent image act as views. Because each crop is fixed by a map πn we chose, SAM masks and DINO features computed once on the parent are pulled back into every view, and any two views read the same parent label wherever they overlap.

2

Network: one pretrained backbone, three decoders

A pretrained VGGT-Omega encoder and aggregator are shared by a geometry decoder, a prompt-conditioned mask decoder and a dense feature decoder. Only the late backbone blocks are adapted, with low-rank adapters.

Network: unposed images pass through an image encoder and aggregator into geometry, mask and feature decoders NETWORK · ONE BACKBONE, THREE DECODERS Unposed images any number of views pretrained VGGT-Omega backbone Image Encoder Aggregator cross-view LoRA Geometry Decoder unchanged Mask Decoder Prompt Encoder mask · points · box Feature Decoder Text Prompt Encoder “sofa” Camera poses & point cloud Prompted masks every view, one pass Dense features multi-view consistent Open-vocabulary 3D masks Mask and feature decoders: prompt projection → cross-view attention → weighted pixel / RGB-guided decoder, on the backbone’s multi-layer tokens.

Network. The geometry decoder keeps the backbone’s camera and point-cloud outputs. The mask decoder turns mask, point or box prompts on any view into masks on every view, and the feature decoder’s dense features are matched against a text prompt for open-vocabulary 3D segmentation.

BibTeX

@misc{ma2026featurepuzzles,
  title  = {Feature Puzzles: Single-Image Supervision for Multi-View 3D-Aware Segmentation},
  author = {Ma, Jiahao and Zhang, Qiang and Liu, Miaomiao and Ahmedt-Aristizabal, David and Nguyen, Chuong},
  year   = {2026},
  url    = {https://featurepuzzles.github.io/}
}