My objective is simple...classify the given sequence of images(video) as either moving or staying still from the perspective of the person inside the car.

Below is an example of the sequence of 12 images animated.

1.Moving from the point of the person inside the car.

Class 0: Moving

2.Staying still from the point of the person inside the car.

Class 1: still

Methods I tried to achieve this:

  1. simple CNN (conv2d) with those 12 images(greyscaled) stacked in the channels dimension.(like Deepmind's DQN). Input to the net is (batch_size, 200, 200, 12)

  2. 3D CNN (conv3d) . Input to the net is (batch_size, 12, 200, 200 ,1)

  3. CNN+LSTM (timedistributed conv2d). Input to the net is (batch_size, 12, 200, 200 ,1)

  4. Late fusion method , which is taking 2 frames from the sequence that are some time steps apart and passing them into 2 CNNs (with same weights) separately and concatenating them in dense layer As mentioned in this paper. This is also like CNN+LSTM without the lstm part. Input to this net is (batch_size, 2, 200, 200, 1) -> the 2 images are first and last frames in the sequence

All the methods I tried failed to achieve my objective. Tried tuning various hyperparameters like learning rate, no of filters in CNN layers etc. and nothing worked.

All the methods had a batch_size of 8 (due to memory constraint) and all images are greyscaled.Used ReLUs for activations and softmax in the last layer. No pooling layer was used.

Any help on why my methods are failing or any pointers to a related work

Many Thanks


CNNs are translation invariant.

You are over complicating the problem. The easiest thing you can do is define a region of interest (ROI) of the hood. In the first case where the car is moving and the reflections are dynamic. In the second case they are static. Just do frame-to-frame image subtraction of the hood. If the vehicle is moving you will have lots of 'edge energy'. If it is not moving it will be just noise.

You can apply the same method to the whole image too. In the static case the image subtraction method may become messy as the clouds are moving along with vehicles and pedestrians. For this case use the image subtractions as input to your method.

Another approach is to run an image stabilization algorithm. OpenCV has one. Look at the transformation outputs (translation, rotation, scale, rigid, similarity, affine, etc.). If you can't make a simple filter on them to determine the two cases, train a classifier.

  • $\begingroup$ Do you mean translation equivariant? pooling layers are responsible for the invariance, right? and I didn't use pooling in my methods. $\endgroup$ – Naveen Apr 16 '18 at 6:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.