I'm trying to predict some properties of videos with Keras using the following rough architecture:
- Feed each frame through the same 2-D convolutional layer.
- Take the outputs of this 2-D convolutional layer and feed them through a 3-D convolutional layer.
There are more hidden layers, but these are the main ones that matter and are messing with my dimensionality. The input of
Conv2D should be
(batch_size, height, width, channels). Each movie has dimensionality
(number_of_frames, height, width, channels). I first had the idea to neglect batching of movies entirely, and treat the batch size and the number of frames equivalently. Then,
Conv2D would output a 4-D tensor, and I would increase its dimensionality to make the ouput a 5-D tensor that I could input into
Conv3D. To do this,
Conv3D could only accept inputs of batch size 1.
I decided against this, because I wanted to batch movies. My current thought is to do this:
conv1 = Conv3D(filters=1, kernel_size =(1,12,12), strides=(1,1,1), data_format='channels_last') conv2 = Conv3D(filters=1,kernel_size=(10,10,10), strides=(1,1,1), data_format='channels_last')
conv1 would represent the 2-D convolutional layer while
conv2 would represent the 3-D convolutional layer. Would this idea work? I figure there is the advantage that I can batch now and when I train the 2-D filter, the same 2-D filter is running over every single movie frame. I'm just worried that the filter in
conv1 will fail to go over certain frames, or it will somehow overlap frames when I want the filter to go over every frame individually.