Is it possible to apply the associative property of the convolution operation when it is followed by a non-linearity?

Question

The associative property of multidimensional discrete convolution says that:

$$Y=(x \circledast h_1) \circledast h_2=x\circledast(h_1\circledast h_2)$$

where $h_1$ and $h_2$ are the filters and $x$ is the input.

I was able to do exploit this property in Keras with Conv2D: first, I convolve $h_1$ and $h_2$, then I convolve the result with $x$ (i.e. the rightmost part of the equation above).

Up to this point, I don't have any problem, and I also understand that convolution is linear.

The problem is when two Conv2D layers have a non-linear activation function after the convolution. For example, consider the following two operations

$$Y_1=\text{ReLU}(x \circledast h_1)$$ $$Y_2=\text{ReLU}(Y_1\circledast h_2)$$

It is possible to apply the associative property if the first or both layers have a non-linear activation function (in the case above ReLU, but it could be any activation function)? I don't think so. Any idea or related paper or some kind of approach?

Answer 1

Yes, when you have non-linearity it is not possible to combine your convolution steps.

However, you can approximate the two layers network with one layer net according to the Universal Approximation Theorem. You will probably need to use something like a knowledge distillation technique to do it. Note that, the theorem doesn't say anything about the number of neurons required or if the learning techniques that we usually use will work well.

Also, $ReLU(x)$ is a linear mapping when $x \geq 0$, so if your input, weight, and biases are $\geq 0$, the net can be exactly modeled with a single layer network.