Is the Machine Learning community going against Occam's razor?

Question

I have been using ML models, for a couple of years, but I am actually in the neuroscience field. In it, mathematical models try to assume the smaller number of things and make hypothesis as simple as possible. This follows Occam's Razor principle of simplicity. My concern is if this is also true for the ML or, more specifically, the Deep Learning community.

I try to briefly illustrate this. When designing DL architectures, I find some of them awfully complicated, so many parameters and layers, hand-crafted loss functions that I wonder if that is really necessary. Of course, some of the problems at hand require a non-trivial solution but sometimes it seems a bit too much. From time to time, you see a paper saying that "we did the same but in a less complicated way". This is cool of course, but I haven't seen many times.

The question is therefore: Should machine learning engineers/researchers put more effort in simplifying architectures? If interpretability is important, the more simple the model the better (i.e. everybody understand how a linear or sigmoid regressor works but a graph-biased-random-walk-based-parametric-dolphin-topologic transformer not so sure...)

P.S. The name of transformer is not real ;)

Answer 1

Regularisation (at least, $L_1$ and $L_2$) can be viewed as an application of Occam's razor. Regularization is widely used in ML and studied in learning theory (see, for example, the structural risk). Another application of Occam's razor is in the AIXI agent.

It seems that models like GPT are going against Occam's razor, but I think they are just brute-force solutions that create only hype - it is obvious to me that with more computation, data and model capacity, a model trained to minimize some objective might perform better, but who cares if the generated text contains fewer errors or "seems to make more sense" if you can't still rely on it to do anything that you really want?

In general, I agree with you that current architectures are more complicated than what they were in the past - but this doesn't necessarily go against Occam's razor, which states (at least in learning theory) that you should choose the simplest hypothesis that is consistent with the data (see also Occam learning), so it doesn't say to just choose the simplest hypothesis.

So, I don't think that all ML community is (necessarily) going against Occam's razor.