1
$\begingroup$

Consider the following neural network with $\ell$ layers.

$$i_0 \rightarrow h_1 \rightarrow h_2 \rightarrow h_3 \cdots \rightarrow h_{\ell-1} \rightarrow o_{\ell} ,$$

where $i, h, o$ stands for input, hidden and output layer respectively.

In general, an input passes from $i_0$ to $o_{\ell}$, which is known as the forward pass. And then the weight updating happens from $o_{\ell}$ to $i_0$ which is called backward pass.

I want to know whether the following mechanism exists in literature assuming that all layers have the same input and output dimensions.

For each iteration

  1. Select a subset $L \subseteq \{0, 1, 2, 3, \cdots, \ell\}$ randomly.
  2. Input passes through layers whose indices are present in $L$ only i.e, forward pass happens by dropping some layers.
  3. Update weights for layers whose indices are in $L$ i.e., update the weights of layers which are participated in step (2).

What is the name of the technique mentioned above, if it is present in literature?

$\endgroup$
4
  • 1
    $\begingroup$ This should be relatively easy to put together in PyTorch or Chainer (any lib that can dope with dynamic graphs - doing this is in e.g. Tensorflow would be much harder). Have you tried it, does it have any useful impact? I'd imagine it may work as a form of regularisation. It also has some relationship to unrolled RNNs, as each layer will tend to work as if it is processing the same kind of input. However, the technique is unlikely to have any references or have been given a name if the results are not good. $\endgroup$ Jul 26 at 7:21
  • 2
    $\begingroup$ The closest technique to what you're proposing that comes to my mind right now is dropout, but, in dropout, we randomly drop some neurons from the neural network, so not full layers. $\endgroup$
    – nbro
    Jul 26 at 12:29
  • $\begingroup$ @NeilSlater I didn't try yet. Need to do. $\endgroup$
    – hanugm
    Jul 26 at 14:37
  • 1
    $\begingroup$ With some playing around, you could probably find some scenarios where this benefits the learning of a model. Unfortunately, I don't think this method will ever outperform dropout as it simply drops too much at once. No definite answer here though, you'll need to do your own testing to evaluate it's effectiveness $\endgroup$
    – Recessive
    Jul 26 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.