2
$\begingroup$

I am currently trying to solve a regression problem using neural networks. I want to detect movement patterns in images over time (video) and output a continuous value. During the training process I noticed a strange behaviour for the validation loss curve and I was wondering if anyone has noticed this kind of periodic pattern on some of their own work. What might cause this?

The model looks like the following: 

- TimeDistributed(Conv2D(32, (3,3)))
- TimeDistributed(Conv2D(16, (3,3)))
- TimeDistributed(Flatten())
- GRU(64, stateful=True)
- Dropout(0.5)
- Dense(64, activation='relu')
- Dense(1)

I trained the model using the mean squared error as the loss function, a batch size of 1 and the AdamOptimizer with an initial learning rate of 10^(-6). Obviously, the loss curve for the training data is not very good, but I am currently just wondering about the pattern of the val_loss. The plots below represent the loss of 65 epochs.

Thanks!

Validation Loss

Training Loss

Edit: The way I try to solve my task relies on a sliding window approach where I try to predict a continuous value for the next second based on the last 20 seconds (400 frames) of the time-series input data. But I don't think this information is needed to solve my initial question since the periodic patterns appear over several epochs (one "peak" for about every 15 epochs) which is strange. Although the stateful-version of the GRU is used (btw: using TensorFlow and Keras), the internal state of the GRU is reset after every epoch to maintain a clean start. The stateful keyword is used to indicate a dependency between batches.

Improve this question
$\endgroup$
2
  • $\begingroup$ To add some more clarity, can you explain more about how you set up your dataset and training pipeline? Perhaps there's a link to a tutorial or a source code you were following? $\endgroup$
    – user3667125
    Dec 18, 2020 at 21:20
  • $\begingroup$ Are you sure that you're going over all your dataset in one epoch? The fact that the loss also has these spikes suggests that you might be seeing new training data you haven't trained on before? $\endgroup$
    – Raphael Lopez Kaufman
    Aug 10, 2022 at 20:10

1 Answer 1

Reset to default
0
$\begingroup$

your model is giving a high loss for start of the video frame , later its loss is getting decreased as it runs through several frames of the "same" video, but again after a new video occurs its loss again peaks to certain point.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .