# Train and Test Accuracy of GRU network not increasing after 2nd epoch

So I´m currently implementing my first neural network using GRUs as a model and Keras as an implementation since it´s pretty highlevel. My problem is about the classification of 8 hour long timeseries with 11 different events with 1 second timesteps or to be more specific: sleep recordings.
Since the GRU can´t handle the whole timeseries at once, I split it in 500 timepoint pieces with 50 seconds (10%) overlap. Also since I don´t have much data, after the training/test split I´m oversampling the train data by duplicating the underrepresented classes up to 10 times. So at the end I get a set of ca. 14.000 training snippets and ca 1.800 to test the model. That to the data.

Next thing is the model, represented with the Keras Code:

verbose, epochs, batch_size = 2, 50, 1200  # 1200 batch_size was the maximum the GPU can handle
n_timesteps, n_features, n_outputs = trainX.shape[1], trainX.shape[2], trainy.shape[1]

model = Sequential()
kernel_regularizer=regularizers.l2(0.01), activation="relu"))
GRU(64, return_sequences=True, go_backwards=False, dropout=0.7, kernel_regularizer=regularizers.l2(0.01),
activation="relu"))
GRU(32, return_sequences=True, go_backwards=False, dropout=0.7, kernel_regularizer=regularizers.l2(0.01),
activation="relu"))
GRU(16, return_sequences=True, go_backwards=False, dropout=0.7, kernel_regularizer=regularizers.l2(0.01),
activation="relu"))

# define custom optimizer with configurable learning rate
sgd = optimizers.SGD(lr=0.1, momentum=0.9, decay=0.0, nesterov=False)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])


So as a short summary:
I am using 4 GRU layers with a "pyramid" shape, so the network has to be more specific towards the end. Also one fully connected layer as some kind of "adapter" and one output layer with the size of my features. I am using SGD as an optimizer.

The results are always pretty bad, here are the loss and accuracy plots of an example run featuring 25 epochs:

Despite the huge dropout each stage, it still seems to be overfitting. Also as you can see, the accuracy of test and train is not changing after the 2nd epoch. The result is this sad looking confusion-matrix, showing only one class beeing detected by the network, which is not even the one with the highest amount of data in the dataset:

[[231   0   0   0   0   0   0   0]
[ 55   0   0   0   0   0   0   0]
[647   0   0   0   0   0   0   0]
[141   0   0   0   0   0   0   0]
[444   0   0   0   0   0   0   0]
[  0   0   0   0   0   0   0   0]
[118   0   0   0   0   0   0   0]
[ 74   0   0   0   0   0   0   0]]


So what are possible approaches in order to:
a) fight the overfitting,
b) just one class beeing detected and
c) fix the stationary accuracy to be growing again?

Since I´m pretty new to neural networks I am thankful for your time and effort!

• I´m alreay using the class_weight="balanced" parameter inside the model.fit() call. Do you mean this with class weighting? And do you have an example with the avg precision for me? – JohnDizzle May 16 '19 at 19:08