I want to implement DSD: Dense-Sparse-Dense training for deep neural networks by Han et al. In short, the paper suggest the following training scheme to improve the network accuracy:

enter image description here

  1. Train as usual till convergence,
  2. prune the network and train with sparsity constraint,
  3. remove the sparsity constraint and let the pruned connections to recover.

My question is about step 2: train with sparsity constraint. The paper mentions training with a binary mask specifying the pruned weights to keep "untouched" so the sparsity constraint is satisfied, however that means implementing a dedicated layer that takes the binary mask as an additional blob and handles it accordingly.

I wonder a simpler approach will give the same result: what if after the pruning step I keep the location of the pruned weights and then use dense training, but after each iteration to zero the originally pruned weights?

The forward path is the same, taking zero weights for the pruned weights anyway. But would it negatively affect the backward path - since the constraint isn't there, or is it equivalent to the formal training scheme?



1 Answer 1


Your approach will give a similar result. The difference with using the constraint is that the constraint allows the network to essentially confirm that it doesn't need the connections before it reallocates them (i.e. if the weight is lessened by the constraint and it results in something not working, then the weight will start increasing).

  • $\begingroup$ Thanks for replying. I don't understand your last sentence (if the weight is lessened...) Could you please elaborate? $\endgroup$
    – rkellerm
    Oct 24, 2017 at 8:50
  • $\begingroup$ I think I might have just misunderstood. I assumed that while running the S step, you were planning to add the sparsity constraint to your re-weighting function to cause low values to be less likely to increase, otherwise setting the values to 0 still allows them to grow during the S step. In my last sentence, when I referred to "constraint" I meant that. $\endgroup$
    – Jeffry
    Dec 7, 2017 at 2:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .