Dust: Pretraining Transformers Without Backpropagation
Q Labs has published Dust, the first zeroth-order method that is competitive with backpropagation for pretraining transformer language models. The paper hit 118 points on Hacker News with 20 comments [1][2].
Backpropagation has been the foundation of deep learning for decades. It requires differentiable functions and produces first-order gradients, and the entire ecosystem of architectures, optimizers, and hardware has been built around this constraint. Dust challenges that by using activation-space perturbation instead: it perturbs activations independently at every token, so each token acts as a virtual population member, and a single forward pass evaluates them all in parallel [1].
The key results are striking. Dust approximates backprop closely at large population sizes and in some settings exceeds it. It is 1'000 to 10'000 times more efficient than weight-space evolution strategies at scale. And counterintuitively, larger models are more population-efficient, not less: a 243M-parameter model outperforms a 120x smaller model at most population sizes [1].
The method works by perturbing activations rather than weights. This is important because it means the perturbations live in the space the model actually processes, not in the enormous weight space where random search is hopeless. Each token in a sequence gets its own perturbation, so a single forward pass through a long sequence effectively evaluates many perturbation samples at once [1].
The research references Sutton's "bitter lesson": general methods that scale with compute eventually win. Backprop is an inductive bias that makes learning efficient in the low-compute regime, but it limits the space of architectures that work. A search-based method like Dust removes the differentiability constraint entirely. You could train non-differentiable architectures, or architectures where backprop gets stuck in bad local minima [1].
The gradient estimates from Dust align well with backprop's as population grows, and they stay aligned at every scale tested up to 1B tokens. This is encouraging for scaling. The authors note that zeroth-order methods are widely believed not to scale to large networks, and their results directly contradict that assumption [1].
This is the kind of research that could matter a lot or very little depending on how it scales. If Dust continues to match or exceed backprop at larger scales, it opens a fundamentally different path for training neural networks. If it stalls, it is still a valuable proof that the backprop monopoly is not as secure as people assumed [1].
Sources:
[1] Q Labs - Dust: Pretraining Transformers Without Backpropagation
[2] Hacker News discussion (118 points, 20 comments)