Does changing the order of the convolution layers in a CNN have any impact?

Question

Could changing the order of convolution layers in a CNN improve accuracy or training time?

Answer 1

Conventionally, CNN layers downsample over and over, which enables them to capture details at different levels of abstractions. Usually, it is observed that the initial layers do nothing more than detecting edges, or filtering color channels; the combinations of these edges are what we perceive as 'features'.

If you reverse the order, you essentially are changing sampling modes down the line. CNNs detect by 'downsampling' the inputs and therefore 'extracting' features.

It may not work as expected!