My objective is simple...classify the given sequence of images(video) as either moving or staying still from the perspective of the person inside the car.
Below is an example of the sequence of 12 images animated.
1.Moving from the point of the person inside the car.
2.Staying still from the point of the person inside the car.
Methods I tried to achieve this:
simple CNN (conv2d) with those 12 images(greyscaled) stacked in the channels dimension.(like Deepmind's DQN). Input to the net is (batch_size, 200, 200, 12)
3D CNN (conv3d) . Input to the net is (batch_size, 12, 200, 200 ,1)
CNN+LSTM (timedistributed conv2d). Input to the net is (batch_size, 12, 200, 200 ,1)
Late fusion method , which is taking 2 frames from the sequence that are some time steps apart and passing them into 2 CNNs (with same weights) separately and concatenating them in dense layer As mentioned in this paper. This is also like CNN+LSTM without the lstm part. Input to this net is (batch_size, 2, 200, 200, 1) -> the 2 images are first and last frames in the sequence
All the methods I tried failed to achieve my objective. Tried tuning various hyperparameters like learning rate, no of filters in CNN layers etc. and nothing worked.
All the methods had a batch_size of 8 (due to memory constraint) and all images are greyscaled.Used ReLUs for activations and softmax in the last layer. No pooling layer was used.
Any help on why my methods are failing or any pointers to a related work