Is my intuition about RNN wrong?

Question

Until today, my intuition about RNN (LSTM/GRU) was that this is some kind of NN that can remember previous inputs.

Consider a task where you need to predict 0 if the previous input was 1. For example: x=[0,1,0,0], y=[1,1,0,1]. We train the model online, i.e. the number of timesteps the RNN sees at a time is equal to one (in Keras you can make an LSTM/GRU cell stateful so that it passes state (memory about previous inputs?) between batches). And the problem is that, in my tests, RNN cells can not cope with this simple task.

I suspect that my intuition about RNN is built wrong. I would appreciate someone could help me with that.

Test script:

import numpy as np
import tensorflow as tf
from random import randrange

batch_size = 1
timesteps = 1
size = 1
units = 16
shape = (timesteps, size)

model = tf.keras.models.Sequential([
  tf.keras.layers.Input(shape = shape, batch_size = batch_size),
  tf.keras.layers.GRU(
    units,
    stateful = True,
    return_sequences = True
  )
])

model.compile(optimizer='adam',
              loss='mean_squared_error')

print(model.summary())

SAME = False

i = 0
prevIsBlack = False
x_batch, y_batch = [], []
x_steps, y_steps = [], []

while True:
  i += 1

  isBlack = randrange(4) == 2

  if SAME: prevIsBlack = isBlack 

  x_train = np.ones(size)   if isBlack else np.zeros(size)
  y_train = np.zeros(units) if prevIsBlack else np.ones(units)

  if not SAME: prevIsBlack = isBlack 

  x_steps.append(x_train)
  y_steps.append(y_train)

  if len(x_steps) < timesteps:
    continue

  x_batch.append(np.stack(x_steps))
  y_batch.append(np.stack(y_steps))

  x_steps, y_steps = [], []

  if len(x_batch) < batch_size:
    continue

  res = model.fit(np.stack(x_batch), np.stack(y_batch), 
    epochs = 1, 
    shuffle = False,
    batch_size = batch_size)

  print(res.history["loss"])

  x_batch = []
  y_batch = []

Answer 1

I got this batch-based GRU to work:

import numpy as np
import tensorflow as tf

units = 2
seq_length = 32

X = (np.random.rand(int(1e4), seq_length, 1) > 0.5).astype(float)
Y = 1 - X

model = tf.keras.models.Sequential([
  tf.keras.layers.Input(shape=X.shape[1:]),
  tf.keras.layers.GRU(units, return_sequences=True),
  tf.keras.layers.Conv1D(1, 1, activation='linear')
])

print(model.summary())

model.compile(optimizer='adam', loss='mean_squared_error')
history = model.fit(X, Y, batch_size=512, epochs=10, verbose=0, validation_split=0.1)

for k in sorted(history.history.keys()):
    print('%15s: %s' % (k, str(np.array(history.history[k][:10]).round(3))))
print()

history = model.fit(X, Y, batch_size=512, epochs=40, verbose=0, validation_split=0.1)

for k in sorted(history.history.keys()):
    print('%15s: %s' % (k, str(np.array(history.history[k][-10:]).round(3))))

It trains for 10 epochs, prints logs, trains for 40 more epochs and prints logs of the last 10 epochs.

Here is an example run:

Model: "sequential_104"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
gru_39 (GRU)                 (None, 32, 2)             30        
_________________________________________________________________
conv1d_68 (Conv1D)           (None, 32, 1)             3         
=================================================================
Total params: 33
Trainable params: 33
Non-trainable params: 0
_________________________________________________________________

           loss: [1.137 0.97  0.832 0.718 0.625 0.546 0.479 0.422 0.372 0.329]
       val_loss: [1.041 0.891 0.767 0.665 0.58  0.508 0.447 0.394 0.348 0.307]

           loss: [0.026 0.023 0.021 0.019 0.017 0.016 0.014 0.013 0.011 0.01 ]
       val_loss: [0.024 0.022 0.02  0.018 0.016 0.015 0.013 0.012 0.011 0.009]

Using 8 units helps it to converge faster, after 10 epochs the loss is about 0.1.

I noticed that your network outputs the GRU's state directly as the predicted value.

Maybe there is a problem since the output dimension doesn't match the target value? There might be some broadcasting going on, which hides the issue. <- edit: my earlier assumption, it isn't correct.

My network's Conv1D converts the GRU's units dimensional output to a N x seq_length x 1 prediction. Adding it to your online version seems to fix it. Although the learning is quite unreliable, maybe due to the batch size of one.

Edit: Aah after reading your code more carefully I noticed that the target predicted value isn't the previous value of the input (a single digit) but this:

y_train = np.zeros(units) if prevIsBlack else np.ones(units)

I don't know a neural network architecture in which we'd measure the loss directly on the RNN's output, it is only used by network to "remember" data or state from the past. But it is then processed further by subsequent layers. First of all the GRU's activation funtion is tanh, so it cannot even produce values outside of the range from -1 to 1.

I hope this clears things up.