In this article here, the writer claims that a new type of neural net is required to deal with data that is both continuous, and also sparsely sampled.
It was my understanding that this was the entire purpose of techniques that use neural nets, to make assumptions about a system with a non-continuous data set.
So why do we need to switch to a non-layered design to deal with these data sets better?
They struggle because if your network have an inductive bias towards modeling datasets which are described with ODEs well, you will learn faster, and with smaller dataset. I think, this what the authors of the original article meant.
In a similar way, CNNs recognize images much better, because their features are translation invariant whereas fully connected net needs to learn to recognize a cat in each different position from scratch.
The question is, "Why do layered neural nets struggle with continuous data?" based on this comment in the Technology Review article.
Data from medical records ... a smattering of measurements at arbitrary intervals. A traditional neural network struggles to handle this. Its design requires it to learn from data with clear stages of observation. Thus it is a poor tool for modeling continuous processes, especially ones that are measured irregularly over time.
— A radical new neural network design could overcome big challenges in AI, Karen Hao, 2018, MIT Technology Review
The article refers to a more technical paper where the NODE approach is proposed.
We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably back-propagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.
— Neural Ordinary Differential Equations (2018, Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, David Duvenaud)
It is correct that largely obscured phenomena can be difficult for any person or machine to properly model. It is important to distinguish between sparsely sampled and irregularly sampled data. Note that neither paper proposes a non-layered design.
Consider these three phenomena.
1. A chaotic property such as heart rate
2. A periodic property such as clock time
3. A trending but noisy property such as glycation
Now consider two sampling methods.
a. Measurement every Monday at noon
b. Measurement whenever a patient comes in for an appointment
We now have six permutations by combining phenomena and sampling methods. Each combination presents modelling difficulties.
1.a. Chaos easily determinable and analyzable but difficult to model because of the number of affecting factors
1.b. Chaos not determinable and difficult to analyze
2.a. Periodicity not determinable because the frequency of sampling is below half the Nyquist criteria and synchronous with the period of the property measured
2.b. Periodicity not determinable because the appointment time may be chaotic or a constant
3.a. Trend is easily determinable but a model must be established for predictability
3.b. Trend is difficult to determine but is possible with a set of models previously demonstrating predictive efficacy
The question indicated that existing artificial networks can make assumptions about a system with a non-continuous data set derived from sampling its behavior. The purpose of a stateful artificial network such as a RNN, LSTM, or GRN is to converge a parametric model using discrete samples, however the effectiveness of most current stateful designs relies on consistency in sampling rate.
To overcome this reliance, the NODE approach uses a plug-in differential equation solver to provide a closed form for this node model.
$$ \dfrac {d h(t)} {d t} = f(h(t), t, \theta) $$
They then state the reasons that computational thrift is achieved.
Not storing any intermediate quantities of the forward pass allows us to train our models with nearly constant memory cost as a function of depth, a major bottleneck of training deep models.
They go on to tout other advantages in terms of computational thrift and then make the key statement with regard to the question.
Continuously-defined dynamics can naturally incorporate data which arrives at arbitrary times.
Note again that it is not specifically sparsity that this new approach intends to overcome but inefficiency created by depth and dimensionality, irregularity of sampling intervals, which is almost unavoidable in low cost medical interventions.
