# Would this neural network have short term memory?

I want to design a NN that can remember it's last 7 actions and use them as inputs. So for example it would be able to store words in it's memory. Therefore if it had a choice of 10 different actions, the number of words it could store is $$10^7$$.

Here is my design:

$$out_{n+1} = f(out_n, in_n)\mathbf{N} + out_n.\mathbf{M}$$

$$action_n = \sigma(\mathbf{N} \cdot out_n)$$

Where $$f$$ represents some layered neural network. Some of the actions would be physical actions and some might be internal (such as thinking of the letter 'C').

Basically I want $$out_n$$ to be an array that keeps the last 6 action values and puts them back in. So $$M$$ will be the matrix:

$$\begin{bmatrix} 0&1&0&0&0&0\\ 0&0&1&0&0&0\\ 0&0&0&1&0&0\\ 0&0&0&0&1&0\\ 0&0&0&0&0&1\\ 0&0&0&0&0&0 \end{bmatrix}$$

i.e. it would drop the 6th item from it's memory.

and $$N$$ would be the vector:

$$\begin{bmatrix} 1&0&0&0&0&0&0 \end{bmatrix}$$

I think this would be equivalent to an equation of the form:

$$out_{n+1}=F(in_n,out_n,out_{n-1},out_{n-2},...,out_{n-6})$$

So I think this would be an advantage over an RNN since this model remembers precisely it's last 6 actions. But would this be better than an RNN or worse? One could increase it's memory to more than 7 quite easily.

I think it's basically the same archececture as an RNN except elinimating a lot of the connections. Is this a new design or a common design?

One problem with this design is that you might also want a memory that is over longer time periods (e.g. for actions that take more than one tick.) But that might be solved by enhancing the archecture.

## 1 Answer

Congrats, you have invented 1d convolution. Convolution combined with RNN would have some advantage over just RNN. Think about the perception field. In this layer, you do aggregate $$6$$ values to one. Imagine two of them - it will be $$36$$ already, etc. But, in the end, you still need RNN at the end to aggregate a variable length to constant length.

• Well that's good! Glad I'm on the right track! (Not sure what you mean at the end about variable lengths). – zooby Jul 27 '19 at 19:14
• @zooby This is not a 1D CNN, its a non differentiable RNN. (actions must be sampled under some categorical distribution based on whats described). The only similarity to a 1d cnn is the sliding window – mshlis Jul 27 '19 at 19:40
• Why is it non-differentiable ? – zooby Jul 27 '19 at 19:42
• do you train with sequences of different lenght, right? also if you put output as input think about output may be wrong so you can consider to force-feeding ( expected data instead of output) – user8426627 Jul 27 '19 at 20:14
• I could be wrong but generally actions are drawn from a distribution (that’s why you show one hot encodingns) and you can’t differentiate through a categorical distrib – mshlis Jul 27 '19 at 20:30