Your Transformer Is Secretly Elastic
Synthetic voice
Simoncini et al. propose a retraining-free method that makes pre-trained transformers elastic, adapting at inference time across computational budgets. They rank every MLP neuron and attention head jointly on a common scale using differentiable soft pruning across multiple sparsity targets, keeping weights frozen. Across 27 vision and text encoders, they report smooth degradation up to 60% sparsity and outperformance of one-shot baselines under deep pruning, with elastic models produced in minutes and construction time scaling sub-linearly with parameter count.
- The method ranks every individual MLP neuron and attention head jointly on a common scale, unlike prior work optimizing one weight per block or head.
- Across 27 encoders from 22M to 8B parameters, the authors report smooth degradation up to 60% sparsity; baselines remain competitive under shallow pruning but the method outperforms them under deep pruning.
- Pruning DINOv3 ViT-H+/16 to 50% sparsity removes 427M parameters with only 0.8% drop in ImageNet-1k linear-probing accuracy.