2
$\begingroup$

The associative property of multidimensional discrete convolution says that:

$$Y=(x \circledast h_1) \circledast h_2=x\circledast(h_1\circledast h_2)$$

where $h_1$ and $h_2$ are the filters and $x$ is the input.

I was able to do exploit this property in Keras with Conv2D: first, I convolve $h_1$ and $h_2$, then I convolve the result with $x$ (i.e. the rightmost part of the equation above).

Up to this point, I don't have any problem, and I also understand that convolution is linear.

The problem is when two Conv2D layers have a non-linear activation function after the convolution. For example, consider the following two operations

$$Y_1=\text{ReLU}(x \circledast h_1)$$ $$Y_2=\text{ReLU}(Y_1\circledast h_2)$$

It is possible to apply the associative property if the first or both layers have a non-linear activation function (in the case above ReLU, but it could be any activation function)? I don't think so. Any idea or related paper or some kind of approach?

$\endgroup$
0
$\begingroup$

Yes, when you have non-linearity it is not possible to combine your convolution steps.

However, you can approximate the two layers network with one layer net according to the Universal Approximation Theorem. You will probably need to use something like a knowledge distillation technique to do it. Note that, the theorem doesn't say anything about the number of neurons required or if the learning techniques that we usually use will work well.

Also, $ReLU(x)$ is a linear mapping when $x \geq 0$, so if your input, weight, and biases are $\geq 0$, the net can be exactly modeled with a single layer network.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thx! I will investigate about "Universal Approximation Theorem", I've never heard. I am trying to optimize cascade convolutions as much as possible since I am implementing convolutional neural networks in an electronic device. $\endgroup$ – Diego Ruiz Jul 1 at 1:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.