Training a Student Expert via Semi-Supervised Foundation Model Distillation

Pardis Taghavi; Tian Liu; Renjie Li; Reza Langari; Zhengzhong Tu

Headline Results

A ~11× smaller student that surpasses its adapted VFM teacher on instance segmentation.

+11.9

maskAP over zero-shot VFM teacher
Cityscapes

+8.6

maskAP over zero-shot VFM teacher
ADE20K

+3.4

maskAP over adapted teacher
Cityscapes (10% labels)

11×

smaller than the 568M-parameter
composite VFM teacher

TL;DR

CAST is a three-stage semi-supervised knowledge distillation framework that compresses large pre-trained vision foundation models (Grounding-DINO + SAM2, 568M params) into a compact 52M-parameter student expert (DINOv2-S + DPT-S + Mask2Former-style decoder) for instance segmentation.

The pipeline runs Teacher Adaptation → Knowledge Transfer → Student Refinement, with an instance-aware pixel-wise contrastive loss that fuses mask and class predictions to mine hard negatives — enforcing sharp inter-instance margins even when only 5–10% of pixels are labeled.

The result: a ~11× smaller student that beats its adapted teacher by +3.4 / +1.5 maskAP on Cityscapes / ADE20K, and runs 4.6× faster at ~3× lower latency.

Overview

Vision foundation models (VFMs) such as Grounding-DINO, SAM2, and DINOv2 deliver strong open-set recognition, but their scale — and their generic pre-training — make them costly to deploy and sub-optimal on specialized downstream tasks like instance segmentation under tight label budgets.

We address this with a stage-wise SSKD pipeline built on two ideas:

Contrastive Calibration. We adapt the VFM teacher via self-training, but inject a pixel-wise contrastive head that sharpens mask boundaries — not just pseudo-labels.
Debiased, Instance-Aware Sampling. During both adaptation and distillation, we mine hard negatives through a joint mask-/class-probability embedding, focusing repulsion on informative inter-instance pairs.

The two ideas compose into three concise stages:

1Teacher Adaptation

Adapt the VFM (Grounding-DINO + SAM2) to the target domain using labels, pseudo-labels, and pixel-wise contrastive regularization. $\mathcal{L}_{\mathrm{sup}} + \mathcal{L}_{\mathrm{semi}} + \mathcal{L}_{\mathrm{pxl}}$.

2Knowledge Transfer

Freeze the adapted teacher and distill into a 52M student (DINOv2-S + DPT-S + Mask2Former-style decoder) using a unified objective over labeled and unlabeled data.

3Student Refinement

Fine-tune the student on labeled data only to correct residual bias from the pseudo-labels — sharpens decision boundaries for the target domain.

A key insight: the same contrastive objective is maintained across both teacher adaptation and student distillation, aligning teacher and student embeddings in a shared instance-aware feature space.

Method: Instance-Aware Pixel-Wise Contrastive Loss

Standard supervised and pseudo-label losses enforce correct masks, but do not explicitly model pixel-level feature relationships. We add a NT-Xent contrastive loss on weak/strong augmented views to better exploit both labeled and unlabeled images.

Pixel-wise contrastive loss

For an anchor pixel $(b, p)$, the positive pair is the same spatial location in the other view; negatives are $R$ hard-negative pixels sampled by our instance-aware sampler:

$$\mathcal{L}_{\mathrm{pxl}} = -\frac{1}{BN}\sum_{b,p}\log\frac{\exp(s^{+}_{b,p})}{\exp(s^{+}_{b,p})+\sum_{r}\exp(s^{-}_{b,p,r})}$$

Debiased, instance-aware negative sampling

We construct a per-pixel sampling distribution by fusing mask and class predictions into a joint pseudo-probability embedding $y[b,p] = \big[P_{m}[b, 1{:}K, p] \,\Vert\, F_{c}[b, p, 1{:}C{+}1]\big]$, where $F_{c}[b,p,c] = \sum_{k} P_{m}[b,k,p]\, P_{c}[b,k,c]$ is the expected class distribution at the pixel.

The dissimilarity score $s^{\mathrm{deb}}\big((b,p),(b',q)\big) = \max\big(0,\, 1 - \langle\tilde{y}[b,p],\, \tilde{y}[b',q]\rangle\big)$ is large when two pixels likely belong to different instances — precisely the pairs that should be repelled. For each anchor we then sample $R$ negatives proportionally to $s^{\mathrm{deb}}$.

Theoretical insight

Under a mild negative-sampling guarantee that the probability a sampled negative is a true inter-instance pixel satisfies $p \gt 0.5$:

Proposition 3.1 (Expected Margin Growth). One gradient update on $\mathcal{L}_{\mathrm{pxl}}$ increases the expected inter-instance margin $\Delta_{\mathrm{emp}}$ by $\varepsilon = \Theta(p \cdot \lambda_{\mathrm{pxl}}) \gt 0$.

This expectation holds even when pseudo-labels are imperfect, provided negatives are sampled by our instance-aware strategy. Empirically we observe $p \gt 0.9$ and an approximately linear increase of $\Delta_{\mathrm{emp}}$ with $\lambda_{\mathrm{pxl}}$.

Empirical Validation

We monitor the empirical margin $\Delta_{\mathrm{emp}} = \mathrm{NegMean} - \mathrm{PosMean}$ and the false-negative rate (FNR, where $p = 1 - \mathrm{FNR}$) every 10k iterations on Cityscapes. The negative-sampling guarantee is satisfied with a wide margin: $p \gt 0.9$ throughout training, and the margin grows approximately linearly with $\lambda_{\mathrm{pxl}}$ — consistent with Proposition 3.1.

**Figure 3.** Empirical margin, FNR, and contrastive loss over training iterations.

Unified Training Framework

Teacher adaptation, student distillation, and student refinement are special cases of a single unified objective:

$$\mathcal{J}(\theta; \mathcal{D}_l, \mathcal{D}_u; \lambda_{\mathrm{semi}}, \lambda_{\mathrm{pxl}}) = \mathcal{L}_{\mathrm{sup}}(\mathcal{D}_l) + \lambda_{\mathrm{semi}}\cdot\mathcal{L}_{\mathrm{semi}}(\mathcal{D}_u) + \lambda_{\mathrm{pxl}}\cdot\mathcal{L}_{\mathrm{pxl}}(\mathcal{D}_l \cup \mathcal{D}_u)$$

Where $\mathcal{L}_{\mathrm{sup}}$ enforces ground-truth supervision on labeled data, $\mathcal{L}_{\mathrm{semi}}$ transfers pseudo-label knowledge on unlabeled data, and $\mathcal{L}_{\mathrm{pxl}}$ imposes pixel-wise contrastive regularization across both sets. The coefficients $\lambda_{\mathrm{semi}}$ and $\lambda_{\mathrm{pxl}}$ balance these signals and turn on/off between stages.

Teacher adaptation — two stages

Fine-tune pretrained VFM weights $\theta_{T}^{0}$ on labeled data only — with contrastive regularization: $\theta_{T}' = \arg\min_{\theta}\, \mathcal{J}(\theta; \mathcal{D}_l, \emptyset; 0, \lambda_{\mathrm{pxl}})$.
Generate pseudo-labels $\hat{y}_{j}^{u} = f_{\theta'_{T}}(x_{j}^{u})$ on unlabeled data, then re-initialize from $\theta_{T}^{0}$ and train on both labeled and pseudo-labeled data: $\theta_{T}'' = \arg\min_{\theta}\, \mathcal{J}(\theta; \mathcal{D}_l, \mathcal{D}_u; 1, \lambda_{\mathrm{pxl}})$.

This two-stage procedure yields a teacher that is better aligned with the target domain and produces pseudo-labels that are both more accurate and more spatially consistent.

Knowledge transfer

With the adapted teacher $\theta_{T}''$ frozen, we train the student on the same unified objective: $\theta_{s}^{*} = \arg\min_{\theta_{s}}\, \mathcal{J}(\theta_{s}; \mathcal{D}_l, \mathcal{D}_u; \lambda_{\mathrm{semi}}, \lambda_{\mathrm{pxl}})$. The contrastive signal continues to align teacher and student embeddings throughout distillation.

Student refinement

Finally, fine-tune the student on labeled data only to reduce residual pseudo-label bias and sharpen decision boundaries: $\theta_{s}^{\dagger} = \arg\min_{\theta}\, \mathcal{J}(\theta; \mathcal{D}_l, \emptyset; 0, 0)$, initialized from $\theta_{s}^{*}$.

Main Results

We evaluate on Cityscapes and ADE20K with 10% labeled data. The teacher is a fused Grounding-DINO-Large + SAM2-L ensemble (568M params); the student is DINOv2-S + DPT-S + Mask2Former-style decoder (52M params, ≈9% of the teacher).

**Table 1.** Main results on Cityscapes and ADE20K (10% labels). * = adapted methods. Blue: best / second-best. Teacher: 568M. Student: 52M.
Method	Data Regime	Cityscapes		ADE20K
		maskAP	maskAP₅₀	maskAP	maskAP₅₀
Teacher Adaptation (568M)
Zero-shot VFM	None (pretrained)	22.0	42.3	8.1	18.2
Supervised fine-tuning	Labeled only	28.7	53.4	14.2	23.5
Self-training*	Labeled+Unlabeled	29.7	54.9	14.6	23.6
Unbiased Teacher*	Labeled+Unlabeled	29.8	54.9	14.8	23.7
CAST (ours)	Labeled+Unlabeled	30.5	56.6	15.2	24.5
Student Distillation (52M)
Supervised fine-tuning	Labeled only	21.1	38.7	13.9	24.2
PAIS	Labeled+Unlabeled	22.9	44.9	10.3	18.3
Guided distillation	Labeled+Unlabeled	30.8	52.9	14.2	23.8
Vemulapalli et al.*	Unlabeled only	24.4	45.6	5.1	9.3
Depth-Guided	Labeled+Unlabeled	30.9	52.9	—	—
S⁴M	Labeled+Unlabeled	33.3	56.7	—	—
CAST (knowledge transfer)	Labeled+Unlabeled	32.2	56.5	16.1	27.4
CAST (student refinement)	Labeled only	33.9	58.7	16.7	28.0

The refined CAST student achieves the best maskAP on both datasets while being roughly 11× smaller than the composite VFM teacher. The teacher-adaptation variant also tops every baseline, suggesting pixel-wise contrastive regularization improves pseudo-label quality and feature discrimination.

Efficiency at a Glance

The CAST student is not only more accurate — it is dramatically more efficient. 90.8% fewer parameters, 77.3% lower FLOPs, 78.1% lower latency, and 4.6× higher FPS on the same hardware.

**Figure 2.** Efficiency comparison (log scale).

−90.8%

Parameters
568M → 52M

−77.3%

FLOPs
880G → 200G

−78.1%

Latency
350.87 ms → 76.92 ms

4.56×

FPS speedup
2.85 → 13.0

Performance-complexity radar chart — **Figure 7.** CAST variants — accuracy vs. efficiency radar (normalized).

Ablation Studies

Loss components

Pseudo-label supervision ($\mathcal{L}_{\mathrm{semi}}$) and pixel-wise contrastive regularization ($\mathcal{L}_{\mathrm{pxl}}$) are complementary: adding both yields the best student (32.2 maskAP on Cityscapes, +11.1 over supervised-only).

**Table 2.** Effect of loss terms on Cityscapes (10% labels).
Method	L_sup	L_semi	L_pxl	Teacher	Student
(a) Sup. only	✓			28.7	21.1
(b) + Pseudo	✓	✓		29.7	30.7
(c) + Pixel loss	✓		✓	29.6	27.5
(d) (b)+(c) — CAST	✓	✓	✓	30.5	32.2

Training stages

All three stages are necessary. Removing teacher adaptation costs −8.2 maskAP; removing student refinement costs −1.7; removing distillation entirely costs −12.8.

**Table 3.** Effect of training stages on Cityscapes (10% labels).
Variant	Teacher Adapt.	Distill.	Student FT	maskAP
Full pipeline	✓	✓	✓	33.9
No student FT	✓	✓		32.2
No teacher adapt.		✓	✓	25.7
Distillation only		✓		23.8
No distill. (Sup.)			✓	21.1

Negative-sampling strategy

Fusing mask and class predictions for negative sampling outperforms either cue alone — confirming that the two sources of information are complementary for identifying informative inter-instance negatives.

**Table 4.** Ablation of negative-sampling strategies on Cityscapes.
Method	maskAP (%)	maskAP₅₀ (%)
Uniform	29.4	50.2
Mask-Only	30.6	55.0
Class-Only	31.1	55.3
Fusion (CAST)	32.2	56.5

Student architecture

DINOv2-S encoder + DPT decoder head offers the best accuracy/efficiency trade-off among the backbones and heads evaluated.

**Table 5.** Student architecture ablation on Cityscapes. Top: encoder (fixed DPT decoder). Bottom: decoder (fixed DINOv2-S encoder).
Encoder	maskAP	maskAP₅₀	Params (M)
ResNet50	25.5	49.3	24
SAM2-S	22.1	39.2	35
DINOv2-S	30.7	54.9	22
Decoder	maskAP	maskAP₅₀	Params (M)
FPN	28.9	52.4	18
DPT	30.7	54.9	22

Scalability with label budget

CAST consistently outperforms prior SSKD baselines across all label fractions on Cityscapes. At 5% labels it achieves 30.7 maskAP — substantially above PAIS (18.0) and Guided Distillation (23.0). At 30% labels it reaches 40.4 maskAP, beating the strongest baseline (S⁴M, 37.8) by +2.6 maskAP.

**Table 6.** Scalability across label fractions on Cityscapes.
Fraction	Teacher Adapt.	Distillation	CAST (refined)	PAIS	Guided dist.	S⁴M
5%	29.4	29.2	30.7	18.0	23.0	30.1
10%	30.5	32.2	33.9	22.9	30.8	33.3
30%	33.3	38.5	40.4	32.8	35.6	37.8

Qualitative Results

Cityscapes — vs. Guided Distillation [3]

Top row: Guided Distillation (baseline). Bottom row: CAST. Note the cleaner instance boundaries, fewer merged boxes, and better separation of adjacent cars and pedestrians.

Qualitative Cityscapes comparison — **Figure 4.** Guided dist. [3] (top) versus CAST (bottom).

Bias reduction across the three stages

Top row: pseudo-labels generated by the adapted teacher. Bottom row: CAST student predictions after distillation + refinement — showing reduced pseudo-label bias and sharper instance boundaries.

**Figure 5.** Qualitative bias reduction in stage-wise distillation.

ADE20K — diverse scenes

Despite training on Cityscapes-style labels, the CAST student generalizes to ADE20K's diverse indoor and outdoor scenes with 100 thing classes.

Qualitative ADE20K results — **Figure 6.** Qualitative results on ADE20K.

Conclusion

We presented CAST, a semi-supervised knowledge distillation pipeline that combines self-training, instance-aware pixel-wise contrastive learning, and supervised refinement to transfer knowledge from large vision foundation models into compact student experts.

The resulting student is ~11× smaller than the teacher while surpassing the adapted teacher by +3.4 maskAP on Cityscapes and +1.5 maskAP on ADE20K. The instance-aware negative-sampling scheme has a theoretical guarantee on the expected inter-instance margin (Proposition 3.1) and is empirically validated with $p \gt 0.9$ throughout training.

Future work includes simplifying the multi-stage pipeline, evaluating on additional domains (medical, remote sensing), and extending the framework to broader efficient perception settings — including open-vocabulary and video scenarios.

BibTeX

@InProceedings{Taghavi_2026_CVPR,
    author    = {Taghavi, Pardis and Liu, Tian and Li, Renjie and Langari, Reza and Tu, Zhengzhong},
    title     = {Training a Student Expert via Semi-Supervised Foundation Model Distillation},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2026},
    pages     = {3620-3630}
}

Acknowledgments

Portions of this research were conducted with the advanced computing resources provided by Texas A&M High Performance Research Computing. We thank the Grounding-DINO, SAM2, DINOv2, and DPT teams for releasing their models and code.

CAST: Training a Student Expert via
Semi-Supervised Foundation Model Distillation

Headline Results

TL;DR

The CAST pipeline: adapt the VFM teacher, distill a compact student, then refine on labels to remove residual pseudo-label bias.