Enabling Validation for Robust Few-Shot Recognition

Overview

Few-Shot Recognition (FSR) suffers from limited task-specific training data and the absence of validation data for model selection and hyperparameter tuning, often leading to overfitting and poor out-of-distribution (OOD) generalization. While retrieved data offer a potential source of validation, directly using their accuracy favors no adaptation dueto their OOD nature.

To resolve this dilemma, we formulate validation as a multi-objective optimization (MOO) problem between performance gains on the few-shot ID data and the performance preservation on the retrieved data. We introduce a simple yet effective validation strategy by choosing an operating point on the Pareto front of the MOO problem. This strategy enables hyperparameter tuning, which had been unrealistically done by using test data, thereby mitigating overfitting and improving generalization. We integrate it into a stage-wise finetuning pipeline, termed Validation-Enabled Stage-wise Tuning (VEST). Extensive experiments on five established benchmarks show that VEST consistently outperforms existing approaches on both ID and OOD test data.

The dilemma of directly adopting retrieved data for validation

Due to data scarcity, no validation data practically exist in FSR. Solely relying on the training accuracy (cf. the dark blue curve on figure below) will select the last checkpoints. They produce notably degraded ID and OOD test accuracy compared with the earlier checkpoints which should preferably be selected. Inspired by the recent work SWAT, which retrieves task-relevant data from open data resources to augment few-shot training images, we exploit such data for validation.

A straightforward idea is to pack the retrieved data as the validation set. However, the retrieved data is OOD compared to the task-specific few-shot training data. Consequently, the finetuned model on the ID data yields degraded performance on the retrieved data. This causes the validation logic to favor the pretrained model without any finetuning, preventing effective finetuning to improve the generalization ability.

Our solution: gF1 score

We present a validation strategy based on ''performance gains'' on the ID training data and the retrieved data. Specifically, for each checkpoint, say at epoch-\(i\), we calculate its training accuracy (\(\text{acc}^i_{ID}\)) and the accuracy on the retrieved data (\(\text{acc}^i_{RT}\)). We place all the checkpoints on a 2D plane with \(x\) and \(y\) specifying the two accuracies. Then, for the \(t^{th}\) checkpoint, we compute its ''performance gains'' on the ID training data as \(\Delta_{ID}^t= \text{acc}^t_{ID} - \min_i(\text{acc}^i_{ID})\), and on the retrieved data as \(\Delta_{RT}^t=\text{acc}^t_{RT}-\min_i(\text{acc}^i_{RT})\). We measure the generalization ability of this checkpoint using their harmonic mean (in spirit of F1 score), dubbed gF1: \begin{equation} \text{gF1}^t = 2 \times \frac{\Delta_{ID}^t \times \Delta_{RT}^t}{\Delta_{ID}^t + \Delta_{RT}^t}. \end{equation}

gF1 enables checkpoint selection

Our validation strategy selects the checkpoint at epoch-3 after fully finetuning the VLM. We verify this selection by comparing these checkpoints w.r.t accuracies on the OOD and ID test data. Results show that the selected checkpoint does generalize robustly well to the test data.

gF1 enables layer selection

We extend our validation strategy to decide how many top layers to partially finetune (PFT) in a Transformer. In practice, it selects the top-4 layers. The resulting checkpoint accuracies for different \(k\) on the ID and OOD test sets demonstrat that our validation strategy effectively determines the top-\(k\) layers to PFT for better robustness and generalization.

VEST: Validation-Enabled Stage-wise Tuning

One-stage training

Our experiments reveal that one-stage finetuning with RA provides limited gains in ID accuracy . We conjecture the reason is that the retrieved examples have

label noises.
distributional shifts compared with task-specific training images.

To address these issues, we adopt a stage-wise finetuning pipeline.

Stage-wise training

Through empirical evaluation of stage-wise approaches, we develop VEST (highlighted in blue) that incorporates RA to PFT the visual encoder with our validation strategy in three stages:

Stage 0: Layer selection for PFT by the proposed validation.
Stage 1: PFT with both ID few-shot training data and the retrieved data.
Stage 2: PFT on the ID few-shot training data.

VEST achieves state-of-the-art performance

We compare representative VLM adaptation methods categorized into Prompt Tuning (PT), Finetuning (FT), Adapter Learning (AL) and Linear Probing (LP). Our method VEST significantly outperforms existing adaptation methods in both ID and OOD accuracy with a moderate number of learned parameters.

BibTeX


      @article{wang2025enabling,
      title={Enabling Validation for Robust Few-Shot Recognition}, 
      author={Wang, Hanxin and Liu, Tian and Kong, Shu},
      journal={arXiv preprint arXiv:2506.04713},
      year={2025}
      }