Length-Bucketed Batching: How to Stop Wasting GPU Cycles on Padding for Small Language Models
The padding problem: why naive batching wastes compute Processing one ticket per forward pass is described in the article as “the single largest source of waste in the whole pipeline. ” The reason is memory bandwidth.