I am a bit confused about NeuralODE and I want to make sure that what I understood so far is correct.

Assume we have (for simplicity) 2 data points $$z_0$$ measured at $$t_0$$ and $$z_1$$ measured at $$t_1$$. Normally (in normal NN approach), one would train a NN to predict $$z_1$$ given $$z_0$$, i.e. $$NN(z_0)=z_1$$. In NeuralODE approach, the goal is to train the NN to approximate a function $$f(z_0)$$ (I will ignore the explicit time dependence) such that given the ODE: $$\frac{dz}{dt}|_{t_0}=f(z_0)$$ which would be approximated as $$\frac{dz}{dt}|_{t_0}=NN(z_0)$$ and solving this using some (non AI based) ODE integrator (Euler's method for example) one gets as the solution for this ODE at time $$t_1$$ something close to $$z_1$$. So basically the NN now approximates the tangent of the function ($$\frac{dz}{dt}$$) instead of the function itself ($$z(t)$$).

Is my understanding so far correct?

So I am a bit confused about the training itself. I understand that they use the adjoint method. What I don't understand is what exactly is being updated. As far as I can see, the only things that are free (i.e. not measured data) are the parameters of the function $$f$$, i.e. the NN approximating it. So one would need to compute $$\frac{\partial loss}{\partial \theta}$$, where $$\theta$$ are the parameters (weights and biases of the network).

Why would I need to compute, for example (as they do in the paper) $$\frac{\partial loss}{\partial z_0}$$? $$Z_0$$ is the input which is fixed, so I don't need to update it. What am I missing here?

Secondly, if what I said in the first part is correct, it seems like in principle one can get great results for a reasonably simple function $$f$$, such as a (for example) 3 layers fully connected NN. So one needs to update the parameters of this NN. On the other hand, ResNets can have tens or hundreds of layers.

Am I missing a step here or is this new approach so powerful that with a lot fewer parameters one can get very good results?

I feel like a ResNet, even with 2 layers, should be more powerful than Euler's Method ODE, as ResNets would allow more freedom in the sense that the 2 blocks don't need to be the same, while in the NeuralODE using Euler's Method one has the same (single) block.

Lastly, I am not sure I understand what do they mean by (continuous) depth in this case. What is the definition of the depth here (I assume it is not just the depth of $$f$$)?

• Can you please provide a link to the paper you're reading about NeuralODE? Are you referring to Neural Ordinary Differential Equations? Furthermore, you're asking too many questions in the same post. Please, create one post for each of your questions, so that to facilitate the life of the answerers. – nbro Jul 11 '19 at 22:14
• @nbro I am sorry. That is the paper (I assumed it is known well enough so I forgot to actually put the link, sorry). Should I delete this post and make several new ones? The reason I did it like this is that the introductory part would be the same for all the questions, so I would have to copy paste that part 3 times for each of my posts. – Alex Marshall Jul 12 '19 at 1:33
• You can edit this post to ask just one question. Then you ask the other questions in other others, unless they are very related to each other (I would say). – nbro Jul 12 '19 at 7:32