# In a convolutional neural network, how is the error delta propagated between convolutional layers?

I'm coding some stuff for CNNs, just relying on numpy (and scipy just for the convolution operation for pure performance reasons).

I've coded a small network consisting of a convolutional layer with several feature maps, the max pooling layer and a dense layer for the output, so far so good, extrapolating the backpropagation from fully connected neural networks was quite intuitive.

But now I'm stuck when several convolutional layers are chained. Imagine the following architecture:

• Output neurons: 10
• Input matrix (I): 28x28
• First convolutional layer (CN1): 3x5x5, stride 1 (output shape is 3x24x24)
• First pooling layer (MP1): 2x2 (output shape is 3x12x12)
• Second convolutional layer (CN2): 3x5x5, stride 1(output shape is 3x8x8)
• Second pooling layer (MP2): 2x2 (output shape is 3x4x4)
• Dense layer (D): 10x48 (fully connected to flattened MP2)

Propagating the error back:

• Error delta in output layer: 10x1 (cost delta)
• Error delta in MP2: 3x4x4 (48x1 unflattened, calculating the error delta for the dense layer as usual)
• Error delta in CN2: 3x8x8 (error delta of MP2 but just upsampled)

How do I keep from here? I don't know how to keep propagating the error to the previous layer, if the error delta in the current one is 3x8x8, and the kernel 3x5x5, performing the convolution between the error delta and the filter for calculating the delta for the previous layer, that gives a 3x4x4 delta.