The channel sizes 32, 128, etc. are used because of memory and efficiency. There is nothing holy about these numbers.
The intuition behind choosing the number of channels is as follows-
The initial layers extract low-level features- they consist of edge detectors, etc. There aren't many such features. So, we won't gain much by adding a lot of filters(of course, if we use a 3x3 filters on an RGB image, we would have 2^27 different filters even if our neurons have only 0 and 1 as their values. However, most of them are quite similar/meaningless for our job). Using a lot of filters might even lead to overfitting.
The latter layers are responsible for detecting more nuanced features, like elbows/nose shape from the lower level features extracted previously. So, we might do better if we increase the number of channels. Also, note that the resultant layers become more and more sparse as we go deeper.
Though it might differ in applications like super resolution image, in general, the number of channels stays the same or increases when we go deeper.
A nice experiment would be to try and increase the number of channels until you get no more benefit from it. I believe there was a paper that did exactly this(please cite it if someone remembers). You could even try to visualise the filters at this stage and see if the filters are similar or not.
Hope it helps.