1
$\begingroup$

If we have a sufficient number of columns (features) and a sufficient number of rows (length or volume) in a dataset that can describe a system without redundancy, we can even train a simple MLP neural network with higher accuracy.

If our dataset is weak or lacks sufficient information to describe a system or data is redundant, then even a powerful neural network like XGBoost cannot achieve anything better.

So, why do highly complex neural networks like AlphaFold exist?

Why didn't they just look for more protein features rather than making the network more complex?

enter image description here

$\endgroup$

1 Answer 1

2
$\begingroup$

So:

  • "even a powerful neural network like XGBoost": XGBoost is not a neural network
  • "we can even train a simple MLP neural network with higher accuracy": higher with respect to what
  • "Why didn't they just look for more protein features rather than making the network more complex?": because data is expensive, and maybe not even plausible to look for more... think about learning how to launch a rocket, you don't want to spend billions in trials and errors to acquire more data

Now, onto your main question "So, why do highly complex neural networks like AlphaFold exist?"... well, just because something "might" work, it does not imply it "will" work.

Indeed, a big enough 1 hidden layer MLP is a universal approximator... is this of any useful? No. It just tells you that eventually, maybe, given that you can solve an NP-hard problem, you might approximate any function.

However, a decision tree or a big enough polynomial are universal approximators, and they have a very simple closed-form

Now, here comes the inductive bias: yes, you can learn anything with an MLP (overfitting), but if you manage to induce some inductive bias in your model, you might hope to get a better convergence, and better generalization

Now, many miss that even stacking layers, thus the reuse of share information, is already an inductive bias, thus the fact that deep networks generalize better than shallow ones

Finally, you have the no-free-lunch theorem, which says roughly "You cannot create a model that works the best in any circumstances... however, you can hope to create a model that is good under a certain distribution of problems

Therefore, even though an MLP might work in any problem, it won't be the best one always, and thus creating highly specialized/carefully crafted models for you problem, might lead you to models that are bad for other problems, but are amazing in your class of problems

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .