# Would this neural network have short term memory?

I want to design a NN that can remember it's last 7 actions and use them as inputs. So for example it would be able to store words in it's memory. Therefore if it had a choice of 10 different actions, the number of words it could store is $$10^7$$.

Here is my design:

$$out_{n+1} = f(out_n, in_n)\mathbf{N} + out_n.\mathbf{M}$$

$$action_n = \sigma(\mathbf{N} \cdot out_n)$$

Where $$f$$ represents some layered neural network. Some of the actions would be physical actions and some might be internal (such as thinking of the letter 'C').

Basically I want $$out_n$$ to be an array that keeps the last 6 action values and puts them back in. So $$M$$ will be the matrix:

$$\begin{bmatrix} 0&1&0&0&0&0\\ 0&0&1&0&0&0\\ 0&0&0&1&0&0\\ 0&0&0&0&1&0\\ 0&0&0&0&0&1\\ 0&0&0&0&0&0 \end{bmatrix}$$

i.e. it would drop the 6th item from it's memory.

and $$N$$ would be the vector:

$$\begin{bmatrix} 1&0&0&0&0&0&0 \end{bmatrix}$$

I think this would be equivalent to an equation of the form:

$$out_{n+1}=F(in_n,out_n,out_{n-1},out_{n-2},...,out_{n-6})$$

So I think this would be an advantage over an RNN since this model remembers precisely it's last 6 actions. But would this be better than an RNN or worse? One could increase it's memory to more than 7 quite easily.

I think it's basically the same archececture as an RNN except elinimating a lot of the connections. Is this a new design or a common design?

One problem with this design is that you might also want a memory that is over longer time periods (e.g. for actions that take more than one tick.) But that might be solved by enhancing the archecture.

Congrats, you have invented 1d convolution. Convolution combined with RNN would have some advantage over just RNN. Think about the perception field. In this layer, you do aggregate $$6$$ values to one. Imagine two of them - it will be $$36$$ already, etc. But, in the end, you still need RNN at the end to aggregate a variable length to constant length.