A ~11× smaller student that surpasses its adapted VFM teacher on instance segmentation.
CAST is a three-stage semi-supervised knowledge distillation framework that compresses large pre-trained vision foundation models (Grounding-DINO + SAM2, 568M params) into a compact 52M-parameter student expert (DINOv2-S + DPT-S + Mask2Former-style decoder) for instance segmentation.
The pipeline runs Teacher Adaptation → Knowledge Transfer → Student Refinement, with an instance-aware pixel-wise contrastive loss that fuses mask and class predictions to mine hard negatives — enforcing sharp inter-instance margins even when only 5–10% of pixels are labeled.
The result: a ~11× smaller student that beats its adapted teacher by +3.4 / +1.5 maskAP on Cityscapes / ADE20K, and runs 4.6× faster at ~3× lower latency.
Vision foundation models (VFMs) such as Grounding-DINO, SAM2, and DINOv2 deliver strong open-set recognition, but their scale — and their generic pre-training — make them costly to deploy and sub-optimal on specialized downstream tasks like instance segmentation under tight label budgets.
We address this with a stage-wise SSKD pipeline built on two ideas:
The two ideas compose into three concise stages:
Adapt the VFM (Grounding-DINO + SAM2) to the target domain using labels, pseudo-labels,
and pixel-wise contrastive regularization. Lsup + Lsemi + Lpxl.
Freeze the adapted teacher and distill into a 52M student (DINOv2-S + DPT-S + Mask2Former-style decoder) using a unified objective over labeled and unlabeled data.
Fine-tune the student on labeled data only to correct residual bias from the pseudo-labels — sharpens decision boundaries for the target domain.
A key insight: the same contrastive objective is maintained across both teacher adaptation and student distillation, aligning teacher and student embeddings in a shared instance-aware feature space.
Standard supervised and pseudo-label losses enforce correct masks, but do not explicitly model pixel-level feature relationships. We add a NT-Xent contrastive loss on weak/strong augmented views to better exploit both labeled and unlabeled images.
For an anchor pixel (b, p), the positive pair is the same spatial location in the other view; negatives are R hard-negative pixels sampled by our instance-aware sampler:
Lpxl = − (1/BN) Σb,p log
exp(s+b,p) / [ exp(s+b,p) + Σr exp(s−b,p,r) ]
We construct a per-pixel sampling distribution by fusing mask and class predictions
into a joint pseudo-probability embedding
y[b,p] = [Pm[b, 1:K, p] | Fc[b, p, 1:C+1]],
where Fc[b,p,c] = Σk Pm[b,k,p] Pc[b,k,c]
is the expected class distribution at the pixel.
The dissimilarity score
sdeb((b,p),(b′,q)) = max(0, 1 − ⟨ỹ[b,p], ỹ[b′,q]⟩)
is large when two pixels likely belong to different instances — precisely the
pairs that should be repelled. For each anchor we then sample R negatives
proportionally to sdeb.
Under a mild negative-sampling guarantee that the probability a sampled negative
is a true inter-instance pixel satisfies p > 0.5:
Proposition 3.1 (Expected Margin Growth). One gradient update on
Lpxl increases the expected inter-instance margin
Δemp by ε = Θ(p · λpxl) > 0.
This expectation holds even when pseudo-labels are imperfect, provided
negatives are sampled by our instance-aware strategy. Empirically we observe
p > 0.9 and an approximately linear increase of Δemp
with λpxl.
We monitor the empirical margin Δemp = NegMean − PosMean and the
false-negative rate (FNR, where p = 1 − FNR) every 10k iterations on Cityscapes.
The negative-sampling guarantee is satisfied with a wide margin:
p > 0.9 throughout training, and the margin grows
approximately linearly with λpxl — consistent with
Proposition 3.1.
Teacher adaptation, student distillation, and student refinement are special cases of a single unified objective:
J(θ; Dl, Du; λsemi, λpxl) =
Lsup(Dl) + λsemi · Lsemi(Du) + λpxl · Lpxl(Dl ∪ Du)
Where Lsup enforces ground-truth supervision on labeled data,
Lsemi transfers pseudo-label knowledge on unlabeled data, and
Lpxl imposes pixel-wise contrastive regularization across both sets.
The coefficients λsemi and λpxl
balance these signals and turn on/off between stages.
θT0 on labeled data only
— with contrastive regularization:
θT′ = argminθ J(θ; Dl, ∅; 0, λpxl).
ŷju = fθ′T(xju)
on unlabeled data, then re-initialize from θT0 and
train on both labeled and pseudo-labeled data:
θT″ = argminθ J(θ; Dl, Du; 1, λpxl).
This two-stage procedure yields a teacher that is better aligned with the target domain and produces pseudo-labels that are both more accurate and more spatially consistent.
With the adapted teacher θT″ frozen, we train the
student on the same unified objective:
θs* = argminθs J(θs; Dl, Du; λsemi, λpxl).
The contrastive signal continues to align teacher and student embeddings throughout distillation.
Finally, fine-tune the student on labeled data only to reduce residual pseudo-label
bias and sharpen decision boundaries:
θs† = argminθ J(θ; Dl, ∅; 0, 0),
initialized from θs*.
We evaluate on Cityscapes and ADE20K with 10% labeled data. The teacher is a fused Grounding-DINO-Large + SAM2-L ensemble (568M params); the student is DINOv2-S + DPT-S + Mask2Former-style decoder (52M params, ≈9% of the teacher).
| Method | Data Regime | Cityscapes | ADE20K | ||
|---|---|---|---|---|---|
| maskAP | maskAP50 | maskAP | maskAP50 | ||
| Teacher Adaptation (568M) | |||||
| Zero-shot VFM | None (pretrained) | 22.0 | 42.3 | 8.1 | 18.2 |
| Supervised fine-tuning | Labeled only | 28.7 | 53.4 | 14.2 | 23.5 |
| Self-training* | Labeled+Unlabeled | 29.7 | 54.9 | 14.6 | 23.6 |
| Unbiased Teacher* | Labeled+Unlabeled | 29.8 | 54.9 | 14.8 | 23.7 |
| CAST (ours) | Labeled+Unlabeled | 30.5 | 56.6 | 15.2 | 24.5 |
| Student Distillation (52M) | |||||
| Supervised fine-tuning | Labeled only | 21.1 | 38.7 | 13.9 | 24.2 |
| PAIS | Labeled+Unlabeled | 22.9 | 44.9 | 10.3 | 18.3 |
| Guided distillation | Labeled+Unlabeled | 30.8 | 52.9 | 14.2 | 23.8 |
| Vemulapalli et al.* | Unlabeled only | 24.4 | 45.6 | 5.1 | 9.3 |
| Depth-Guided | Labeled+Unlabeled | 30.9 | 52.9 | — | — |
| S4M | Labeled+Unlabeled | 33.3 | 56.7 | — | — |
| CAST (knowledge transfer) | Labeled+Unlabeled | 32.2 | 56.5 | 16.1 | 27.4 |
| CAST (student refinement) | Labeled only | 33.9 | 58.7 | 16.7 | 28.0 |
The refined CAST student achieves the best maskAP on both datasets while being roughly 11× smaller than the composite VFM teacher. The teacher-adaptation variant also tops every baseline, suggesting pixel-wise contrastive regularization improves pseudo-label quality and feature discrimination.
The CAST student is not only more accurate — it is dramatically more efficient. 90.8% fewer parameters, 77.3% lower FLOPs, 78.1% lower latency, and 4.6× higher FPS on the same hardware.
Pseudo-label supervision (Lsemi) and pixel-wise contrastive
regularization (Lpxl) are complementary: adding
both yields the best student (32.2 maskAP on Cityscapes, +11.1 over supervised-only).
| Method | Lsup | Lsemi | Lpxl | Teacher | Student |
|---|---|---|---|---|---|
| (a) Sup. only | ✓ | 28.7 | 21.1 | ||
| (b) + Pseudo | ✓ | ✓ | 29.7 | 30.7 | |
| (c) + Pixel loss | ✓ | ✓ | 29.6 | 27.5 | |
| (d) (b)+(c) — CAST | ✓ | ✓ | ✓ | 30.5 | 32.2 |
All three stages are necessary. Removing teacher adaptation costs −8.2 maskAP; removing student refinement costs −1.7; removing distillation entirely costs −12.8.
| Variant | Teacher Adapt. | Distill. | Student FT | maskAP |
|---|---|---|---|---|
| Full pipeline | ✓ | ✓ | ✓ | 33.9 |
| No student FT | ✓ | ✓ | 32.2 | |
| No teacher adapt. | ✓ | ✓ | 25.7 | |
| Distillation only | ✓ | 23.8 | ||
| No distill. (Sup.) | ✓ | 21.1 |
Fusing mask and class predictions for negative sampling outperforms either cue alone — confirming that the two sources of information are complementary for identifying informative inter-instance negatives.
| Method | maskAP (%) | maskAP50 (%) |
|---|---|---|
| Uniform | 29.4 | 50.2 |
| Mask-Only | 30.6 | 55.0 |
| Class-Only | 31.1 | 55.3 |
| Fusion (CAST) | 32.2 | 56.5 |
DINOv2-S encoder + DPT decoder head offers the best accuracy/efficiency trade-off among the backbones and heads evaluated.
| Encoder | maskAP | maskAP50 | Params (M) |
|---|---|---|---|
| ResNet50 | 25.5 | 49.3 | 24 |
| SAM2-S | 22.1 | 39.2 | 35 |
| DINOv2-S | 30.7 | 54.9 | 22 |
| Decoder | maskAP | maskAP50 | Params (M) |
| FPN | 28.9 | 52.4 | 18 |
| DPT | 30.7 | 54.9 | 22 |
CAST consistently outperforms prior SSKD baselines across all label fractions on Cityscapes. At 5% labels it achieves 30.7 maskAP — substantially above PAIS (18.0) and Guided Distillation (23.0). At 30% labels it reaches 40.4 maskAP, beating the strongest baseline (S4M, 37.8) by +2.6 maskAP.
| Fraction | Teacher Adapt. | Distillation | CAST (refined) | PAIS | Guided dist. | S4M |
|---|---|---|---|---|---|---|
| 5% | 29.4 | 29.2 | 30.7 | 18.0 | 23.0 | 30.1 |
| 10% | 30.5 | 32.2 | 33.9 | 22.9 | 30.8 | 33.3 |
| 30% | 33.3 | 38.5 | 40.4 | 32.8 | 35.6 | 37.8 |
Top row: Guided Distillation (baseline). Bottom row: CAST. Note the cleaner instance boundaries, fewer merged boxes, and better separation of adjacent cars and pedestrians.
Top row: pseudo-labels generated by the adapted teacher. Bottom row: CAST student predictions after distillation + refinement — showing reduced pseudo-label bias and sharper instance boundaries.
Despite training on Cityscapes-style labels, the CAST student generalizes to ADE20K's diverse indoor and outdoor scenes with 100 thing classes.
We presented CAST, a semi-supervised knowledge distillation pipeline that combines self-training, instance-aware pixel-wise contrastive learning, and supervised refinement to transfer knowledge from large vision foundation models into compact student experts.
The resulting student is ~11× smaller than the teacher while
surpassing the adapted teacher by +3.4 maskAP on Cityscapes
and +1.5 maskAP on ADE20K. The instance-aware negative-sampling scheme
has a theoretical guarantee on the expected inter-instance margin
(Proposition 3.1) and is empirically validated with p > 0.9 throughout
training.
Future work includes simplifying the multi-stage pipeline, evaluating on additional domains (medical, remote sensing), and extending the framework to broader efficient perception settings — including open-vocabulary and video scenarios.
@InProceedings{Taghavi_2026_CVPR,
author = {Taghavi, Pardis and Liu, Tian and Li, Renjie and Langari, Reza and Tu, Zhengzhong},
title = {Training a Student Expert via Semi-Supervised Foundation Model Distillation},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2026},
pages = {3620-3630}
}Portions of this research were conducted with the advanced computing resources provided by Texas A&M High Performance Research Computing. We thank the Grounding-DINO, SAM2, DINOv2, and DPT teams for releasing their models and code.