3
$\begingroup$

Imagine I have a tensorflow CNN model with good accuracy but maybe too many filters:

  • Is there a way to determine which filters have more impact in output? I think it should be possible. At least, if a filter A has a 0, that only multiples the output of a filter B, then filter B is not related to filter A. In particular, I'm thinking in 2d data where 1 dimension is time-related and the other feature related (like one-hot char).

  • Is there a way to eliminate the less relevant filters from a trained model, and leave the rest of the model intact?

  • Is it useful or there are better methods?

$\endgroup$
4
$\begingroup$

NOTE: All the observations and results are from the paper The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks.

To answer your questions one by one:

  • Yes there are ways to determine which filters have more impact on the output. Its a very naive way but works very good in practice. Filters with small weights impact output less (according to empirical evidence), which basically means neurons whose weights lie in the switching region i.e ~$0$ in ReLu and ~$-1$ to $1$ (say) have less impact on final output.
  • Yes, just eliminating these lower weight filters eliminate the unnecessary noise and indecisiveness introduced by these filters and suprisingly makes the model perform better (observed empirically).
  • The concept a relatively old paradigm but has been a new twist by the simplicity of the method of elimination of unnecessary weights in the aforementioned paper and thus winning it the best paper award in ICLR 2019.

TL;DR: Eliminating unnecessary weights makes the model perform better than the original model.

Also here is the TensorFlow code.

$\endgroup$
  • 1
    $\begingroup$ Wondeful! Thanks. $\endgroup$ – ESL Sep 9 at 18:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.