2
$\begingroup$

I found this question very interesting, and this is a follow up on it.

Presumably, we'd want all the filters to converge towards some complementary set, where each filter fills as large a niche as possible (in terms of extracting useful information from the previous layer), without overlapping with another filter.

A quick thought experiment tells me (please correct me if I'm wrong) that if two filters are identical down to maximum precision, then without adding in any other form of stochastic differentiation between them, their weights will be updated in the same way at each step of gradient descent during training. Thus, it would be a very bad idea to initialise all filters in the same way prior to training, as they would all be updated in exactly the same way (see footnote 1).

On the other hand, a quick thought experiment isn't enough to tell me what would happen to two filters that are almost identical, as we continue to train the network. Is there some mechanism causing them to then diverge away from one another, thereby filling their own "complementary niches" in the layer? My intuition tells me that there must be, otherwise using many filters just wouldn't work. But during back-propagation, each filter is downstream, and so they don't have any way of communicating with one another. At the risk of anthropomorphising the network, I might ask "How do the two filters collude with one another to benefit the network as a whole?"


Footnotes:

  1. Why do I think this? Because the expession for the partial derivative of the $k$th filter weights with respect to the cost $\partial W^k/\partial C$ will be identical for all $k$. From the perspective of back-propagation, all paths through the filters look exactly the same.
$\endgroup$
1
$\begingroup$

Yes, your thought experiment is correct, and the concept is known as breaking the symmetry. This is why biases can be initialized to $0$ (bias initialization doesn't matter), but weights should be randomly initialized to different numbers -- to break the symmetry. Otherwise, if not, the network will function as if it has $n-1$ filters (or however many filters that are unique) instead of the full $n$ filters.

As for your main question, if two filters are initialized to very similar values, they may branch out as long as that is what minimizes the training loss. There is no collusion or coordination going on; each filter updates completely independently. You can even freeze all the other filters and only perform gradient descent on one filter at a time. Each filter just follows the direction of their gradient to minimize the training loss.

Consider the backprop equations as defined by this online book:

enter image description here

The gradient of the current layer's weights depends on

  1. The future layers' weights, errors, and activation function's derivatives
  2. The current layer's activation function's derivative, and
  3. The previous layer's outputs.

Each weight in the layer (i.e. each filter in the layer) looks at different parts of these three components (indexed by $j$ and $k$ in equation $BP4$). It is this different perspective that allows them to update their gradients in different directions, even if their initial weights are very similar to each other. Note that it is possible that they end up with the same gradient, but it is very unlikely.

$\endgroup$
2
  • $\begingroup$ Thanks. I'm glad to have my initial thought experiment sense checked. As for the second part, I'm aware of the equations and the mechanisms of back propagation. And it sounds like you conclude with something like: "nothing guarantees that the filters diverge. it's possible that they randomly converge". Cool, this may be the answer. It's weird for me coming from a physics background though - because in mathematics and nature systems similar to this (very hand wavy I know) tend to either converge or diverge. $\endgroup$ Dec 31 '20 at 9:23
  • $\begingroup$ Even though the equations and mechanisms don't express explicit interaction between filters, I can't help but think that the system as a whole might implicitly do it. I'll leave this question open for a little while to see if anyone else has something to say. $\endgroup$ Dec 31 '20 at 9:25
0
$\begingroup$

Here I am just trying to simplify what @user3667125 already said uses math arguments

Say we have a cost function $J(x, y; F(\cdot; \Theta))$ which regards training a NN $F(\cdot, \Theta)$ with $x$ input and $y$ expected output

Gradient descent tells us how to upgrade each $\theta_{i} \in \Theta$ and it is

$$ \theta_{i}(t+1) = \theta_{i}(t) - \alpha \frac{\partial J(x,y; \Theta)}{\partial \theta_{i}} $$

with $t$ training time

So let's focus on a specific component of the NN $f(x; \theta)$ and from its perspective, we can say the computation is

$$ h(f(g(x); \theta), y) $$

with

  • $h(x,y)$ representing all the subsequent components + loss function
  • $g(x)$ representing all the previous computation
  • $x$ input
  • $y$ expected output

so its update is

$$ \Delta \theta(t) = \frac{\partial h(f(g(x(t), \theta), y(t)))}{\partial \theta} $$

with

  • $x(t)$ the concrete input at $t$ time
  • $y(t)$ expected output at $t$ time
  • $\theta(t)$ the concrete value of the parameter at $t$ time

Applying the chain rule we have

$$ \Delta \theta(t) = h'(f(g(x(t), \theta(t))), y(t)) f'(g(x(t)), \theta(t)) $$

so the gradient observed by a certain parameter $\theta$ depends on

  • $\theta(t)$ the current value of the parameter
  • $x(t)$ the current input and more specifically by $g(x(t))$ the processing of this input by the previous part of the NN
  • $y(t)$ the expected output
  • $h'(\cdot)$ the gradient of the subsequent part of the network

So even if we have 2 weights with the same value $\theta_{i}(t) = \theta_{j}(t) \quad i \neq j$ at a certain point in training time, they can still see different gradients since

  • the upstream processing $g(x(t))$ can be different
  • the gradient backpropagating from the downstream processing $h'(\cdot, y(t))$ can be different
$\endgroup$
1
  • $\begingroup$ Thanks for the breakdown. I think that two filters in the same layer will have the same downstream and upstream values and gradients. Therefore if we have two filters with all same values they will evolve in the same way. $\endgroup$ Dec 31 '20 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.