[D] Packing multiple shorter training examples in to single sequence in LM pretraining
I've come across several papers where authors mention they, for reasons of computational efficiency, will pack multiple shorter training examples together in a single sequence.
Example from "Scaling Instruction-Finetuned Language Models" (Chung et al, 2022):
We use packing (Raffel et al., 2020) to combine multiple training examples into a single sequence, separating inputs from targets using an end-of-sequence token. Masking is applied to prevent the tokens from attending to others across the packed example boundary.
I'm curious to understand how this is actually done in practice. Seeing as multiple separate masks are involved, I'd think one would need to loop over them all and repeat (?) the matrix multiplication several times? Is there some built in functionality in Pytorch and other frameworks to deal with a situation like this with multiple masks?
Thankful if someone could share and explain, or link to an implementation of input packing.
Posted Jan 15, 2023, 6:45 AM UTC · edited
Maybe just by setting the attention masks such that the model can't attend outside a subsequence. If you think of the attention mask as an adjacency matrix in a graph then you would essentially have two disconnected subsets of nodes in the resulting graph
I understand how that is done for two (sub)sequences. But here we have potentially more than two. The issue is that masks are binary. If we have more than 2 examples being packed into a single sequence, one would need several sets of masks, each applied independently. It doesn't necessarily seem very efficient to construct several separate masks and repeatedly apply them. It's not entirely obvious to me how one would efficiently implement input packing and attention masking.
After spending some more time searching for a reference I found a paper explaining some of the available options for input packing:
(Krell et al., 2021)
You do not need several sets of masks. Each entry in the mask is a binary indicator of whether a query token can attend to a key token (a connectivity matrix).
You can either attend to another token or you can't, it doesn't matter if you're dealing with subseq 1 or 8, each subseq is concerned only with itself, and treats all other subseqs as "other" or unconnected.
Assuming a 1 indicates that two tokens can attend to each other, and you start with an all zero matrix (thinking about MLM setting for simplicity sake, but also holds for autoregressive models).
If your first seq extends from index 0 to 10, set mask[0:10,0:10] to 1. Your second seq is from 10 to 20, so you set the block mask[10:20,10:20] to 1, and so on.
Thanks, that makes sense. For some reason I was stuck thinking there would be cross contamination if I had all masks in the same matrix. I.e. that the mask indices between example boundaries weren't disjoint. But with your example I realize there's no such problem.