What should be taken as random variables in the distributions of datasets?

Question

Consider the following two paragraphs taken from the paper titles Generative Adversarial Nets by Ian J. Goodfellow et.al

#1: Abstract

We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1.......

#2: Excerpt from Introduction

The promise of deep learning is to discover rich, hierarchical models that represent probability distributions over the kinds of data encountered in artificial intelligence applications, such as natural images, audio waveforms containing speech, and symbols in natural language corpora.

here we saw the word distribution thrice. It is very common to encounter the phrase data distribution in machine learning papers. For the types of data we use in artificial intelligence, there can be infinite random samples. We collect some instances and form a dataset, based on which we try to get the data distribution.

Even most of the literature uses the phrases data distribution or probability distribution. I personally never came across papers that explicitly talk about the random vector for the probability distribution under consideration. What can be the reason for it? Why won't research papers give the list of random variables for the distribution they are discussing? Is it immaterial or is it obvious from the context?

---

Note: For example, if our discussion is about a 'human face images' dataset and if we use the phrase 'data distribution' then should I assume random variables to be 'pixels' of the image or high-level features like 'eyes, ears, etc.,' or some other? Or is it immaterial? If immaterial, then what should I need to perceive about probability distribution? Should I imagine it as an arbitrary random vector?

Answer 1

The main reason of using the term data distribution over the random variable is to note for the intrinsic relationship the different data samples have with each other.

It is a mathematical way of saying this data is almost random (consider the average pixel on a human face dataset) but it has some underlaying relationship (the concept of human faces).

This is also helpful to describe that different images can be similar looking but represent different underlaying concepts. For example: consider 2 dataset: human faces, $x_{human}$, and animal faces, $x_{animal}$. Although most of the images might look similar at a pixel level: similar colors, similar textures, similar backgrounds... The underlaying concept depicted in them differs: human faces vs animal faces.

So, how do you represent that mathematically?: with two overlapping distributions of different shape.

This basically mean: "even though the images might seem random, might overlap in some domains (color, textures, histogram...) there is an intrinsic difference between the two" and this difference (the shape of the distribution) is the target intrinsic concept that the neural network will have to learn.

In the specific case of GANs the left distribution would be the "Generated Faces" and this distribution will try to fit the right distribution that would be "Real Faces".