# Why do layered neural nets struggle with continous data?

In this article here, the writer claims that a new type of neural net is required to deal with data that is both continuous, and also sparsely sampled.

It was my understanding that this was the entire purpose of techniques that use neural nets, to make assumptions about a system with a non-continuous data set.

So why do we need to switch to a non-layered design to deal with these data sets better?

• If time (or intervals between measurements) matter then neural nets don't work as well because Neural nets excel at finding patterns. Neural nets tend to find patterns that have the same sequence, regardless of interval between inputs. If a sequence of inputs is exactly the same taken at fixed intervals or sparse intervals, the neural net will tend to find the patterns as the same; which may not be desirable. If a person's health is evaluated at 30 then 50, you'd want different results than evaluations between 30 and 31, assuming the 31 year old's health now matches the 50 year old's health. – Dunk Dec 18 '18 at 20:41
• I only posted because you got no answers and I thought I could shed a little bit more light on what the article said but did a poor job of conveying in an easily understandable way. My comment isn't worthy of being posted as an answer to your question but thanks for the offer. – Dunk Dec 19 '18 at 0:51

They struggle because if your network have an inductive bias towards modeling datasets which are described with ODEs well, you will learn faster, and with smaller dataset. I think, this what the authors of the original article meant.

In a similar way, CNNs recognize images much better, because their features are translation invariant whereas fully connected net needs to learn to recognize a cat in each different position from scratch.

The question is, "Why do layered neural nets struggle with continuous data?" based on this comment in the Technology Review article.

Data from medical records ... a smattering of measurements at arbitrary intervals. A traditional neural network struggles to handle this. Its design requires it to learn from data with clear stages of observation. Thus it is a poor tool for modeling continuous processes, especially ones that are measured irregularly over time.

— A radical new neural network design could overcome big challenges in AI, Karen Hao, 2018, MIT Technology Review

The article refers to a more technical paper where the NODE approach is proposed.

We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably back-propagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.

— Neural Ordinary Differential Equations (2018, Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, David Duvenaud)

It is correct that largely obscured phenomena can be difficult for any person or machine to properly model. It is important to distinguish between sparsely sampled and irregularly sampled data. Note that neither paper proposes a non-layered design.

Consider these three phenomena.

1. A chaotic property such as heart rate
2. A periodic property such as clock time
3. A trending but noisy property such as glycation

Now consider two sampling methods.

a. Measurement every Monday at noon
b. Measurement whenever a patient comes in for an appointment

We now have six permutations by combining phenomena and sampling methods. Each combination presents modelling difficulties.

1.a. Chaos easily determinable and analyzable but difficult to model because of the number of affecting factors
1.b. Chaos not determinable and difficult to analyze
2.a. Periodicity not determinable because the frequency of sampling is below half the Nyquist criteria and synchronous with the period of the property measured
2.b. Periodicity not determinable because the appointment time may be chaotic or a constant
3.a. Trend is easily determinable but a model must be established for predictability
3.b. Trend is difficult to determine but is possible with a set of models previously demonstrating predictive efficacy

The question indicated that existing artificial networks can make assumptions about a system with a non-continuous data set derived from sampling its behavior. The purpose of a stateful artificial network such as a RNN, LSTM, or GRN is to converge a parametric model using discrete samples, however the effectiveness of most current stateful designs relies on consistency in sampling rate.

To overcome this reliance, the NODE approach uses a plug-in differential equation solver to provide a closed form for this node model.

$$\dfrac {d h(t)} {d t} = f(h(t), t, \theta)$$

They then state the reasons that computational thrift is achieved.

Not storing any intermediate quantities of the forward pass allows us to train our models with nearly constant memory cost as a function of depth, a major bottleneck of training deep models.

They go on to tout other advantages in terms of computational thrift and then make the key statement with regard to the question.

Continuously-defined dynamics can naturally incorporate data which arrives at arbitrary times.

Note again that it is not specifically sparsity that this new approach intends to overcome but inefficiency created by depth and dimensionality, irregularity of sampling intervals, which is almost unavoidable in low cost medical interventions.