How would DeepMind's new differentiable neural computer scale?

Question

DeepMind just published a paper about a differentiable neural computer, which basically combines a neural network with a memory.

The idea is to teach the neural network to create and recall useful explicit memories for a certain task. This complements the abilities of a neural network well, because NNs only store knowledge implicitly in the weights and the information used to work on a single task is only stored in the activation of the network and degrades quickly the more information you add. (LSTMs are one try to slow down this degradation of short term memories, but it still happens.)

Now, instead of keeping the necessary information in the activation, they presumably keep the addresses of memory slots for specific information in the activation, so these should also be subject to degradation.

Why would this approach scale? Shouldn't a somewhat higher number of task-specific information once again overwhelm the network's capability of keeping the addresses of all the appropriate memory slots in its activation?

Answer 1

Examining the architecture of the DNC indeed shows many similarities to the LSTM. Consider the diagram in the DeepMind article that you linked to:

Compare this to the LSTM architecture (credit to ananth on SlideShare):

There are some close analogs here:

• Much like the LSTM, the DNC will perform some conversion from input to fixed-size state vectors (h and c in the LSTM)
• Likewise, the DNC will perform some conversion from these fixed-size state vectors to potentially arbitrarily-lengthed output (in the LSTM we repeatedly sample from our model until we are satisfied/the model indicates we are done)
• The forget and input gates of the LSTM represent the write operation in the DNC ('forgetting' is essentially just zeroing or partially zeroing memory)
• The output gate of the LSTM represents the read operation in the DNC

However, the DNC is definitely more than an LSTM. Most obviously, it utilizes a larger state which is discretized (addressable) into chunks; this allows it to make the forget gate of the LSTM more binary. By this I mean that the state is not necessarily eroded by some fraction at every time step, whereas in the LSTM (with the sigmoid activation function) it necessarily is. This might reduce the problem of catastrophic forgetting that you mentioned and thus scale better.

The DNC is also novel in the links that it uses between memory. However, this might be a more marginal improvement on the LSTM than it seems if we re-imagine the LSTM with complete neural networks for each gate instead of just a single layer with an activation function (call this a super-LSTM); in this case, we can actually learn any relationship between two slots in memory with a sufficiently powerful network. While I don't know the specifics of the links that DeepMind is suggesting, they imply in the article that they are learning everything just by backpropagating gradients like a regular neural network. Therefore whatever relationship they are encoding in their links should theoretically be learnable by a neural network, and so a sufficiently powerful 'super-LSTM' should be able to capture it.

With all that being said, it is often the case in deep learning that two models with the same theoretical capability for expressiveness perform vastly different in practice. For example, consider that a recurrent network can be represented as a huge feed-forward network if we just unroll it. Similarly, the convolutional network isn't better than a vanilla neural network because it has some extra capacity for expressiveness; in fact, it is the constraints imposed on its weights that makes it more effective. Thus comparing the expressiveness of two models isn't necessarily a fair comparison of their performance in practice, nor an accurate projection of how well they will scale.

One question I have about the DNC is what happens when it runs out of memory. When a classical computer runs out of memory and another block of memory is requested, programs start crashing (at best). I'm curious to see how DeepMind plans to address this. I assume it will rely on some intelligent cannibalization of memory currently in use. In some sense computers currently do this when an OS requests that applications free up non-critical memory if memory pressure reaches a certain threshold.