Is an optimization algorithm equivalent to a neural network?


Recurrent neural networks (RNNs) are Turing complete (that is, they can simulate any Turing machine), which means that, in theory, they can execute any algorithm that a Turing machine can, so it can also run any optimization algorithm that can be run on a Turing machine (e.g. gradient descent). See this good answer to the question Meaning (and proof) of "RNN can approximate any algorithm" for a more exhaustive discussion of this fact.

A neural network is a model: it represents some (fixed) function, given some fixed weights (which can be thought of as the state of the NN). However, as opposed to most optimization algorithms, the weights or parameters of a NN can be changed by an optimization algorithm (like gradient descent), but this will arguably no more be the same NN. So, in theory, you could have an RNN that approximates gradient descent and back-propagation, so that a neural network trains another neural network.

An optimization algorithm is usually not trained to approximate any continuous function. It is just a possibly parametrized program that runs on a Turing machine. However, note again that a specific optimization algorithm corresponds to a specific NN, so, in this sense, a specific optimization algorithm is equivalent to some NN.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.