I have been using ML models, for a couple of years, but I am actually in the neuroscience field. In it, mathematical models try to assume the smaller number of things and make hypothesis as simple as possible. This follows Occam's Razor principle of simplicity. My concern is if this is also true for the ML or, more specifically, the Deep Learning community.
I try to briefly illustrate this. When designing DL architectures, I find some of them awfully complicated, so many parameters and layers, hand-crafted loss functions that I wonder if that is really necessary. Of course, some of the problems at hand require a non-trivial solution but sometimes it seems a bit too much. From time to time, you see a paper saying that "we did the same but in a less complicated way". This is cool of course, but I haven't seen many times.
The question is therefore: Should machine learning engineers/researchers put more effort in simplifying architectures? If interpretability is important, the more simple the model the better (i.e. everybody understand how a linear or sigmoid regressor works but a graph-biased-random-walk-based-parametric-dolphin-topologic transformer not so sure...)
P.S. The name of transformer is not real ;)