Do different architectures really make a difference or is it just a matter of the training process?

Question

I was wondering which influence different architectures for deep learning truly have on the performance. Of course, substantial changes in the paradigms we use when building neural networks (such as convolutions or transformers opposed to simple feed forward networks) bring new possibilities for the network to extract features.

But latest when it comes to adjustments in the architectures many papers propose (like adding one layer here, a residual connection there...), does this really make a difference? In my opinion, with enough training and the right choice of hyperparameters, vanilla architectures should be capable to reproduce results that were generated with models that relied on adjustments of the architecture.

So the question is: Suppose method A (vanilla architecture) produces a result of 80% accuracy, but method B (small adjustments to this architecture) achieved 83% accuracy on the same dataset - is this truly significantly because of the architectural change or would method A be capable of also achieving 83%, if trained and tuned for a longer time? Is it really worth to finetune/enhance the few big vanilla architectures out there and to create variants of them? Or should research focus more on tuning hyperparameters and training procedures to improve results for specific tasks?

Answer 1

The answer might be based on opinion, but yes it matters. The concept is called Exploration vs Exploitation.

Consider that you are standing on the top of a hill and you need to go down to the foot. There are practically infinite routes, so you can try some routes here and there. You have two methods here:

• Exploration: just try until you find the optimal route. Now you see where this is going: there is no guarantee that it works and we don't have an infinite amount of resources
• Exploitation: it is when you find a "good enough" route, but you continue to go down that route and ignore the rest.

Balancing Exploration and Exploitation is the way forward. Going back to your example, Exploration here means tuning till...the end, and Explotation here means "test new methods and if it works, we use it". I believe we tend to pick the second, since it is at least some progress rather than waiting forever.