Despite the problem being very simple, I was wondering why an LSTM network was not able to converge to a decent solution.
import numpy as np import keras X_train = np.random.rand(1000) y_train = X_train X_train = X_train.reshape((len(X_train), 1, 1)) model= keras.models.Sequential() model.add(keras.layers.wrappers.Bidirectional(keras.layers.LSTM(1, dropout=0., recurrent_dropout=0.))) model.add(keras.layers.Dense(1)) optimzer = keras.optimizers.SGD(lr=1e-1) model.build(input_shape=(None, 1, 1)) model.compile(loss=keras.losses.mean_squared_error, optimizer=optimzer, metrics=['mae']) history = model.fit(X_train, y_train, batch_size=16, epochs=100)
After 10 epochs, the algorithm seems to have reached its optimal solution (around
1e-4 RMSE), and is not able to improve further the results.
A simple Flatten + Dense network with similar parameters is however able to achieve 1e-13 RMSE.
I'm surprised the LSTM cell does not simply let the value through, is there something I'm missing with my parameters? Is LSTM only good for classification problems?