In many applications and domains, computer vision, natural language processing, image segmentation, and many other tasks, neural networks (with a certain architecture) are considered to be by far the most powerful machine learning models.
Nevertheless, algorithms, based on different approaches, such as ensemble models, like random forests and gradient boosting, are not completely abandoned, and actively developed and maintained by some people.
Do I correctly understand that the neural networks, despite being very flexible and universal approximators, for a certain kind of tasks, regardless of the choice of the architecture, are not the optimal models?
For the tasks in computer vision, the core feature, which makes CNNs superior, is the translational invariance and the encoded ability to capture the proximity properties of an image or some sequential data. And the more recent transformer models have the ability to choose which of the neighboring data properties is more important for its output.
But let's say I have a dataset, without a certain structure and patterns, some number of numerical columns, a lot of categorical columns, and in the feature space (for classification task) the classes are separated by some nonlinear hypersurface, would the ensemble models be the optimal choice in terms of performance and computational time?
In this case, I do not see a way to exploit CNNs or attention-based neural networks. The only thing that comes to my head, in this case, is the ordinary MLP. It seems that, on the one hand, it would take significantly more time to train the weights than the trees from the ensemble. On the other hand, both kinds of models work without putting prior knowledge to data and assumptions on its structure. So, given enough amount of time, it should give a comparable quality.
Or can there be some reasoning that neural network is sometimes bound to give rather a poor quality?