I learned from this post about the so-called bit memory:

They froze its self-attention and feed-forward layers and, in separate copies, fine-tuned peripheral layers on each on a wide range of tasks: Bit memory (memorizing strings of bits), Bit XOR (performing logical operations on pairs of strings of bits), ListOps (parsing and performing mathematical operations), MNIST, CIFAR-10 (classification of images), CFAR-10 LRA (classification of flattened, greyscale images), and remote homology detection (predicting what kind of protein structure an amino acid is part of).

I wonder what the "bit memory" task is? Is it an identity function as described in this post? Or the memory network?


1 Answer 1


Read the paper. It tells you. (page 3)

Bit memory. Similar to the task proposed by Miconi et al. (2018), we consider a bit memory task where the model is shown 5 bitstrings each of length 1000. Afterwards, the model is shown a masked version of one of the bitstrings, where each bit is masked with probability 0.5, and the model is tasked with producing the original bitstring. The bitstrings are broken up into sequences of length 50, so that the models are fed 120 tokens of dimension 50.

It wasn't immediately clear to me whether they train the model on the bitstrings (in which case 5 is quite low and a Transformer model is not required) or just present them to it when running the model. Figure 4 (page 8) makes it clear that it's the latter, as the original bitstrings and the query both appear on the same attention plot.

i.e. the input is something like:


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .