3
$\begingroup$

As computers are getting bigger better and faster, the concept of what constitutes a single datum is changing.

For example, in the world of pen-and-paper, we might take readings of temperature over time and obtain a time-series in which an individual datum is a time, temperature pair. However, it is now common to desire classifications of entire time-series, in the context of which our entire temperature time-series would be but a single data point in a data set consisting of a great number of separate time-series. In image processing, an $(x,y,c)$ triple is not a datum, but a whole grid of such values is a single datum. With lidar data and all manner of other fields things that were previously considered a dataset are now best thought of as a datum.

What is the term for datasets that are themselves composed of datasets?

The term "metadata" is occupied, I should think.

Are there any papers that talk about this transition from datasets of data to datasets of datasets? And what the implications are for data scientists and researchers?

Improve this question
$\endgroup$
2
  • $\begingroup$ PLEASE NOTE: I am not asking "what are the implications of this?", which would be an open-ended question. I am asking for references that discuss the phenomenon. The existence or non-existence of such references is not open-ended. $\endgroup$
    – Him
    Commented Sep 24, 2019 at 19:36
  • $\begingroup$ An interesting topic, you might be interested in latent variable models: sciencedirect.com/topics/neuroscience/latent-variable-models Hope this answers something, I look forward to a discussion. $\endgroup$
    – cashew
    Commented Sep 25, 2019 at 0:28

1 Answer 1

Reset to default
1
$\begingroup$

I don't think that this is anything new. Let's use your example of classifying an entire time series, say predicting word 1 vs word 2 for speech recognition. We can write out the data as a data frame like we would do with any other multivariate data: observations at time 1, time 2, etc as the predictors and the classification label as the response variable.

Each observation is a vector of the values at particular times for your subject, plus the label--no different than any other multivariate data. Sure, there might be special dependence structure because of the time series nature of your data, but you can still write it as a multivariate problem.

Okay, let's say that you hit the speech signal with a wavelet transform, resulting in an image-looking spectrogram of 2D data. Then just consider each "pixel" (time-frequency pair) to be a variable in your multivariate problem, along with the classification label. This is some kind of bijection between an $m\times n$ matrix and $\mathbb{R}^{n\times m}$.

You can extend this idea to 3D or 4D data (or higher), too. Just unwrap the high-dimension tensor in some kind of map $T^{m\times n \times \dots} \rightarrow \mathbb{R}^{m\times n \times \dots}$.

Improve this answer
$\endgroup$
9
  • $\begingroup$ "Each observation is a vector of the values at particular times for your subject, plus the label" this is not how most learning methods treat the problem. Suppose that I want to classify a time-series. Your model has been trained to take on points $(t,y,l)$ where $l$ is a label. The constraint that all of the points in the time-series I want to classify have an identical label $l$ at least shows that naive application of datasets-of-data algorithms is non-trivial in this context. $\endgroup$
    – Him
    Commented Sep 26, 2019 at 19:31
  • $\begingroup$ @Scott I assume that $t$ is time and $y$ is the value of the time series at time $t$. Let's also assume $T$ time samples. Then you could write your observation as $(y_{1},\dots,y_{T}, l)$. Then you would have one row of this $T+1$-dimensional observation per labeled subject. $\endgroup$
    – Dave
    Commented Sep 26, 2019 at 19:38
  • $\begingroup$ This assumes that all of my time-series are equal in length. Of course, I agree with you that analysis of datasets of datasets is not impossible, and I agree with you that many situations can be handled with various tricks to arrange the data into familiar fashions. To bring the conversation back to my original question, which is less about "how to handle the situation" and more about "how to refer to the situation"... The vectorizing trick in your immediately previous comment: Under what situations and contexts could you employ it? Only time-series? $\endgroup$
    – Him
    Commented Sep 26, 2019 at 19:48
  • $\begingroup$ @Scott Uneven length time series wreck the data frame idea but not the idea of seeing the time series as a vector. For instance, represent different length time series in a graphical format (SPARQL-style): label as a node -> value at time 1 -> value at time 2 -> ... Do this with multiple branches from the label node if you have several time series (such as in a spectrogram). I see this method working for images and language, in addition to time series. $\endgroup$
    – Dave
    Commented Sep 26, 2019 at 19:58
  • $\begingroup$ "I see this method working for images and language, in addition to time series" Neat. How would you succinctly denote this class of problems? Note that the descriptor you use can't actually include all time-series problems, since simple prediction on a time-series does not fall into this class of problem. What is this class of problem? Are there any sources that attempt to generalize your vectorization technique to this whole class? Or other analysis techniques? $\endgroup$
    – Him
    Commented Sep 26, 2019 at 21:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .