When do the ensemble methods beat neural networks?

Question

In many applications and domains, computer vision, natural language processing, image segmentation, and many other tasks, neural networks (with a certain architecture) are considered to be by far the most powerful machine learning models.

Nevertheless, algorithms, based on different approaches, such as ensemble models, like random forests and gradient boosting, are not completely abandoned, and actively developed and maintained by some people.

Do I correctly understand that the neural networks, despite being very flexible and universal approximators, for a certain kind of tasks, regardless of the choice of the architecture, are not the optimal models?

For the tasks in computer vision, the core feature, which makes CNNs superior, is the translational invariance and the encoded ability to capture the proximity properties of an image or some sequential data. And the more recent transformer models have the ability to choose which of the neighboring data properties is more important for its output.

But let's say I have a dataset, without a certain structure and patterns, some number of numerical columns, a lot of categorical columns, and in the feature space (for classification task) the classes are separated by some nonlinear hypersurface, would the ensemble models be the optimal choice in terms of performance and computational time?

In this case, I do not see a way to exploit CNNs or attention-based neural networks. The only thing that comes to my head, in this case, is the ordinary MLP. It seems that, on the one hand, it would take significantly more time to train the weights than the trees from the ensemble. On the other hand, both kinds of models work without putting prior knowledge to data and assumptions on its structure. So, given enough amount of time, it should give a comparable quality.

Or can there be some reasoning that neural network is sometimes bound to give rather a poor quality?

Answer 1

Speed:
A classic random forest is O(n) to train and O(1) to run while a feedforward neural network is something like O(n^5) to train and O(n^4) to run, so for many tasks the CART ensemble can train fast and run fast.

Robustness (kinda):
A random forest tends to be natively robust, while GBM and neural networks tend to not be as robust. There are tweaks to loss functions, and to training sampling, that can make the network less not-robust, but that isn't the same as being robust. Dropout doesn't bootstrap the domain or target, only the structure.

Basis:
The CART presumes hard edges. The neural network presumes "continuous". They are decent at handling hard-edged surfaces, as long as you use them well.

https://stats.stackexchange.com/questions/164048/can-a-random-forest-be-used-for-feature-selection-in-multiple-linear-regression/164250#164250

References:
https://lunalux.io/computational-complexity-of-neural-networks/

Answer 2

This is a great question. Unfortunately the answer is that this is still not very well understood and is an active area of research.

I think doing justice to this problem is beyond the scope of an answer here. Instead I will refer you to some recent research papers that attempt to answer this question.

https://arxiv.org/abs/2106.03253

https://arxiv.org/abs/2207.08815