4
$\begingroup$

Does it make sense to use Batch Normalization in Deep (stacked) or/and Sparse Autoencoders? I cannot find any resources for that, so is it safe to assume that since it works for other DNNs it will also make sense to use it and will offer benefits on training AEs?

$\endgroup$
2
$\begingroup$

The term Batch Normalization is both

  • A system reliability choice (in terms of convergence) and
  • an execution strategy.

Batching is generally the process of focusing on process P with source data S to produce result R under conditions that are favorable in terms of timing, data availability, and resource utilization, such as these.

  • P is requires nontrivial time and computing resource and there is a window of time when it can be done on hardware H with minimal impact on other operations
  • R is needed by multiple other processes P2, P3, ... and draws on a set of additional inputs I2, I3, ... so that completing P in larger groups is efficient

One cannot draw a general conclusion about the value of batching that will apply to all specific installations of hardware, networking, real time demands, functional intention, and ML topology. Also, beware of equating deep with stacked. Depth may have multiple meanings now, but was originally used for the dimension describing the number of layers in a multi-layer perceptron, with width being the number of artificial neurons in each layer.

The term stacking can be ambiguously related to depth. Consider that process flows through artificial neural nets has often been represented graphically as signal propagation from left to right, as is the convention in electrical engineering and text in western languages.

We then have two directions for stacking that means two different things.

  • Stacking in the same direction as signal flow, which indicates a sequence of operations on a vector, matrix, cube, or hyper-cube
  • Stacking in the tangential direction with respect to signal flow, which indicates a connectivity between items in the stack that is related to supervision, collaboration, or competition

Whether Sparse Autoencoders or other elements of DNNs should be fronted with batch normalization has two forks to the answer.

  • Is normalization helpful in reducing either signal saturation in gradient calculations during propagation or overflow conditions?
  • If that answer is, "Yes," will batching the normalization have some value for reasons like the ones described above?
$\endgroup$
  • $\begingroup$ Complicated but helpful reply, thanks. My question was more like, since stacking AEs means train layer by layer, will BN offer something? For example, BN shows benefits on training deep feedforward networks by reducing the covariance shift of the hidden units, leading to faster training. Would it make sense on a shallow network? If so, I'd assume it'd also make sense in stacked layers. $\endgroup$ – TasosGlrs Jul 24 '18 at 11:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.