Is there anything that ensures that convolutional filters end up different from one another?

Question

I found this question very interesting, and this is a follow up on it.

Presumably, we'd want all the filters to converge towards some complementary set, where each filter fills as large a niche as possible (in terms of extracting useful information from the previous layer), without overlapping with another filter.

A quick thought experiment tells me (please correct me if I'm wrong) that if two filters are identical down to maximum precision, then without adding in any other form of stochastic differentiation between them, their weights will be updated in the same way at each step of gradient descent during training. Thus, it would be a very bad idea to initialise all filters in the same way prior to training, as they would all be updated in exactly the same way (see footnote 1).

On the other hand, a quick thought experiment isn't enough to tell me what would happen to two filters that are almost identical, as we continue to train the network. Is there some mechanism causing them to then diverge away from one another, thereby filling their own "complementary niches" in the layer? My intuition tells me that there must be, otherwise using many filters just wouldn't work. But during back-propagation, each filter is downstream, and so they don't have any way of communicating with one another. At the risk of anthropomorphising the network, I might ask "How do the two filters collude with one another to benefit the network as a whole?"

---

Footnotes:

1. Why do I think this? Because the expession for the partial derivative of the $k$th filter weights with respect to the cost $\partial W^k/\partial C$ will be identical for all $k$. From the perspective of back-propagation, all paths through the filters look exactly the same.

Answer 1

Yes, your thought experiment is correct, and the concept is known as breaking the symmetry. This is why biases can be initialized to $0$ (bias initialization doesn't matter), but weights should be randomly initialized to different numbers -- to break the symmetry. Otherwise, if not, the network will function as if it has $n-1$ filters (or however many filters that are unique) instead of the full $n$ filters.

As for your main question, if two filters are initialized to very similar values, they may branch out as long as that is what minimizes the training loss. There is no collusion or coordination going on; each filter updates completely independently. You can even freeze all the other filters and only perform gradient descent on one filter at a time. Each filter just follows the direction of their gradient to minimize the training loss.

Consider the backprop equations as defined by this online book:

The gradient of the current layer's weights depends on

1. The future layers' weights, errors, and activation function's derivatives
2. The current layer's activation function's derivative, and
3. The previous layer's outputs.

Each weight in the layer (i.e. each filter in the layer) looks at different parts of these three components (indexed by $j$ and $k$ in equation $BP4$). It is this different perspective that allows them to update their gradients in different directions, even if their initial weights are very similar to each other. Note that it is possible that they end up with the same gradient, but it is very unlikely.

Answer 2

Here I am just trying to simplify what @user3667125 already said uses math arguments

Say we have a cost function $J(x, y; F(\cdot; \Theta))$ which regards training a NN $F(\cdot, \Theta)$ with $x$ input and $y$ expected output

Gradient descent tells us how to upgrade each $\theta_{i} \in \Theta$ and it is

$$ \theta_{i}(t+1) = \theta_{i}(t) - \alpha \frac{\partial J(x,y; \Theta)}{\partial \theta_{i}} $$

with $t$ training time

So let's focus on a specific component of the NN $f(x; \theta)$ and from its perspective, we can say the computation is

$$ h(f(g(x); \theta), y) $$

with

• $h(x,y)$ representing all the subsequent components + loss function
• $g(x)$ representing all the previous computation
• $x$ input
• $y$ expected output

so its update is

$$ \Delta \theta(t) = \frac{\partial h(f(g(x(t), \theta), y(t)))}{\partial \theta} $$

with

• $x(t)$ the concrete input at $t$ time
• $y(t)$ expected output at $t$ time
• $\theta(t)$ the concrete value of the parameter at $t$ time

Applying the chain rule we have

$$ \Delta \theta(t) = h'(f(g(x(t), \theta(t))), y(t)) f'(g(x(t)), \theta(t)) $$

so the gradient observed by a certain parameter $\theta$ depends on

• $\theta(t)$ the current value of the parameter
• $x(t)$ the current input and more specifically by $g(x(t))$ the processing of this input by the previous part of the NN
• $y(t)$ the expected output
• $h'(\cdot)$ the gradient of the subsequent part of the network

So even if we have 2 weights with the same value $\theta_{i}(t) = \theta_{j}(t) \quad i \neq j$ at a certain point in training time, they can still see different gradients since

• the upstream processing $g(x(t))$ can be different
• the gradient backpropagating from the downstream processing $h'(\cdot, y(t))$ can be different