I am not a researcher, but I am curious to know what considerations are relevant to take into account during research for the invention of a new neural network model, and what relevant knowledge researchers typically possess in the area.
And an accompanying question: is a background in neuroscience relevant to such an investigation?
In my experience researchers typically base their architectures on previously identified successful architectures and principles. That is to say published methods that have been successful in practice on similar tasks to the current one. This can be followed back to very early networks like the Perceptron which took a lot of inspiration from existing successful mathematical techniques.
This is not to say that researchers do not draw from neuroscience or from broader knowledge to influence their decisions. I believe most research is guided by more abstract intuitions derived from broader experience. Some researchers are very strongly influenced by neuroscience and I understand that some try to replicate behaviours seen in the brain. The similar structure and functions of the brain will surely make knowledge of it useful. Some major advances may have been motivated by direct analogy to the brain, although I am not aware if that is the case. But, on balance, having zero knowledge of neuroscience is not an impediment to undertaking successful research into neural architectures. Having zero knowledge of existing successful techniques is a far greater impediment.
Having a good level of mathematical understanding is also very useful. I will take perhaps a slight risk of going beyond my knowledge and suggest that university-level knowledge of mathematics is also more beneficial than knowledge of neuroscience. I think that is true for research so far, maybe it will change in the future. All these comments are based only on my personal experience and not on any formal research. Also, note my bias: I have minimal knowledge of neuroscience myself.
