I'm trying to predict some properties of videos with Keras using the following rough architecture:

  1. Feed each frame through the same 2-D convolutional layer.
  2. Take the outputs of this 2-D convolutional layer and feed them through a 3-D convolutional layer.

There are more hidden layers, but these are the main ones that matter and are messing with my dimensionality. The input of Conv2D should be (batch_size, height, width, channels). Each movie has dimensionality (number_of_frames, height, width, channels). I first had the idea to neglect batching of movies entirely, and treat the batch size and the number of frames equivalently. Then, Conv2D would output a 4-D tensor, and I would increase its dimensionality to make the ouput a 5-D tensor that I could input into Conv3D. To do this, Conv3D could only accept inputs of batch size 1.

I decided against this, because I wanted to batch movies. My current thought is to do this:

conv1 = Conv3D(filters=1, kernel_size =(1,12,12), strides=(1,1,1), data_format='channels_last')
conv2 = Conv3D(filters=1,kernel_size=(10,10,10), strides=(1,1,1), data_format='channels_last')

conv1 would represent the 2-D convolutional layer while conv2 would represent the 3-D convolutional layer. Would this idea work? I figure there is the advantage that I can batch now and when I train the 2-D filter, the same 2-D filter is running over every single movie frame. I'm just worried that the filter in conv1 will fail to go over certain frames, or it will somehow overlap frames when I want the filter to go over every frame individually.

  • $\begingroup$ Could you be more explicit about the properties you want to predict? $\endgroup$ – Edoardo Guerriero Apr 16 '20 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.