How does the memory augmented neural network (MANN) work? How can I make a simple MANN with a vanilla neural network especially without a recurrent network?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
If you're referring to the Meta-Learning with Memory-Augmented Neural Networks paper, the premise is that In some cases, we wish to perform meta-learning when we are only provided with small (or no) data. For example - in one/few-shot learning we are expected to provide a label with only a few tagged samples. To do so, we use two techniques
- External memory module - we initialize a container of knowledge that is called upon to respond to challenging circumstances
- Label shuffling - labels are presented one time-step after their corresponding sample, which helps with dealing with simply learning the mapping $sample\rightarrow label$ without generalizing. More specifically, we feed the network with $(x_1, null),(x_2,y_1),...,(x_{t+1},y_t)$
In addition, samples are shuffled across different datasets (samples from different datasets may appear in the same sequence). This encourages the network to use the memory module and extract the relevant label once the corresponding sample is provided.
In terms of actual implementation, memory retrieval is based on a general similarity between vectors (I suggest going over the paper for more details).