The term Batch Normalization is both
- A system reliability choice (in terms of convergence) and
- an execution strategy.
Batching is generally the process of focusing on process P with source data S to produce result R under conditions that are favorable in terms of timing, data availability, and resource utilization, such as these.
- P is requires nontrivial time and computing resource and there is a window of time when it can be done on hardware H with minimal impact on other operations
- R is needed by multiple other processes P2, P3, ... and draws on a set of additional inputs I2, I3, ... so that completing P in larger groups is efficient
One cannot draw a general conclusion about the value of batching that will apply to all specific installations of hardware, networking, real time demands, functional intention, and ML topology. Also, beware of equating deep with stacked. Depth may have multiple meanings now, but was originally used for the dimension describing the number of layers in a multi-layer perceptron, with width being the number of artificial neurons in each layer.
The term stacking can be ambiguously related to depth. Consider that process flows through artificial neural nets has often been represented graphically as signal propagation from left to right, as is the convention in electrical engineering and text in western languages.
We then have two directions for stacking that means two different things.
- Stacking in the same direction as signal flow, which indicates a sequence of operations on a vector, matrix, cube, or hyper-cube
- Stacking in the tangential direction with respect to signal flow, which indicates a connectivity between items in the stack that is related to supervision, collaboration, or competition
Whether Sparse Autoencoders or other elements of DNNs should be fronted with batch normalization has two forks to the answer.
- Is normalization helpful in reducing either signal saturation in gradient calculations during propagation or overflow conditions?
- If that answer is, "Yes," will batching the normalization have some value for reasons like the ones described above?