I want to implement DSD: Dense-Sparse-Dense training for deep neural networks by Han et al. In short, the paper suggest the following training scheme to improve the network accuracy:
- Train as usual till convergence,
- prune the network and train with sparsity constraint,
- remove the sparsity constraint and let the pruned connections to recover.
My question is about step 2: train with sparsity constraint. The paper mentions training with a binary mask specifying the pruned weights to keep "untouched" so the sparsity constraint is satisfied, however that means implementing a dedicated layer that takes the binary mask as an additional blob and handles it accordingly.
I wonder a simpler approach will give the same result: what if after the pruning step I keep the location of the pruned weights and then use dense training, but after each iteration to zero the originally pruned weights?
The forward path is the same, taking zero weights for the pruned weights anyway. But would it negatively affect the backward path - since the constraint isn't there, or is it equivalent to the formal training scheme?