0
$\begingroup$

I had an idea for a layer, but I'm not sure if it exists already and/or if it's implementable in tensorflow.

I would like to have a layer that is similar to a Dense layer, in the way that it's connected to every nodes of the previous layer, but only K connections can be "active" (weights >= 0), and all the other ones will have their weights equals to 0

Which ones of those K connections will be learned by the network, not chosen randomly when building it.

So for example, in a Dense layer that goes from 1000 to 2 features, I would have 10002 + 2 = 2002 parameters that can be different from 0. In my layer, if I chose K = 10, I will have only 210+2 = 22 parameters that can be different from 0.

Any way I can do that and train a network with it?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ perhaps Dropout is what you're referring to $\endgroup$
    – Hadar Sharvit
    Jul 8, 2022 at 19:36
  • $\begingroup$ Dropout is a regularization method used during the training to prevent overfitting. It randomly choses which connection to "drop", but should not be used at inference time. I am looking to have my trained network sparse :) $\endgroup$
    – Adrien Nivaggioli
    Jul 11, 2022 at 7:44

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .