I was wondering which influence different architectures for deep learning truly have on the performance. Of course, substantial changes in the paradigms we use when building neural networks (such as convolutions or transformers opposed to simple feed forward networks) bring new possibilities for the network to extract features.

But latest when it comes to adjustments in the architectures many papers propose (like adding one layer here, a residual connection there...), does this really make a difference? In my opinion, with enough training and the right choice of hyperparameters, vanilla architectures should be capable to reproduce results that were generated with models that relied on adjustments of the architecture.

So the question is: Suppose method A (vanilla architecture) produces a result of 80% accuracy, but method B (small adjustments to this architecture) achieved 83% accuracy on the same dataset - is this truly significantly because of the architectural change or would method A be capable of also achieving 83%, if trained and tuned for a longer time? Is it really worth to finetune/enhance the few big vanilla architectures out there and to create variants of them? Or should research focus more on tuning hyperparameters and training procedures to improve results for specific tasks?


1 Answer 1


The answer might be based on opinion, but yes it matters. The concept is called Exploration vs Exploitation.

Consider that you are standing on the top of a hill and you need to go down to the foot. There are practically infinite routes, so you can try some routes here and there. You have two methods here:

  • Exploration: just try until you find the optimal route. Now you see where this is going: there is no guarantee that it works and we don't have an infinite amount of resources
  • Exploitation: it is when you find a "good enough" route, but you continue to go down that route and ignore the rest.

Balancing Exploration and Exploitation is the way forward. Going back to your example, Exploration here means tuning till...the end, and Explotation here means "test new methods and if it works, we use it". I believe we tend to pick the second, since it is at least some progress rather than waiting forever.

  • $\begingroup$ Alright, but why does finetuning architectures then often only brings an improvement of few percent? Shouldn't it yield bigger performance boosts? I often get the feeling that those results are still not truly significant in terms of the statistical range... So maybe it's just luck and not really because of the change in architecture? $\endgroup$
    – convaldo
    Jan 26, 2023 at 8:34
  • 1
    $\begingroup$ It is not statistical significance in terms of...statistical. In terms of money, it is quite a lot if we are talking in millions or billions. $\endgroup$ Jan 26, 2023 at 15:35
  • $\begingroup$ Oh I see what you are saying! That's an interesting point, indeed $\endgroup$
    – convaldo
    Jan 26, 2023 at 15:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .