I am working on dynamical systems using Optimal Control theory and trying to find the connection between this field and Machine Learning. Consider a simple 2-layer Neural Network (NN) where the activation function is considered as $y = x+x^2 $ (I intentionally ignored bias term and supposed this activation function only for illustration purpose). So, if the output of the first layer is calculated as $y_1=w_ix_i+w_i^2x_i^2$, then the input to output mathematical relation can be given as:


By expanding the terms, it results in:


But we know that there could be four parameters $\alpha_1,\alpha_2,\alpha_3,\alpha_4$ as


in the input to output relation, but the NN structure suggests only three parameters $w_i,w_1,w_o$ to be trained. So, the degrees of freedom is less than the actual potential of the given structure.

Will this lead to any inefficiency in NN model? Could you please suggest some references which mathematically studies the NN structure?

  • $\begingroup$ It is also possible to ask questions about "Optimal Control Theory" on our Operations Research site. --- If posting the same question on two sites be certain to make a note of that in the question, so people know that there might be efforts put forth elsewhere; so they don't duplicate the work. --- Here you'll get a better Machine Learning answer, while over there you'll get better optimization. $\endgroup$
    – Rob
    Commented Jun 24, 2022 at 13:49


You must log in to answer this question.