How can we "draw i.i.d" from any probability distribution?

Question

Consider the following paragraph from 2 Learning in High Dimensions in from of the paper titled Geometric Deep Learning Grids, Groups, Graphs, Geodesics, and Gauges

Supervised machine learning, in its simplest formalisation, considers a set of $N$ observations $D = \{(x_i, y_i)\}_{i=1}^{N}$ drawn i.i.d. from an underlying data distribution $P$ defined over $\mathcal{X} \times \mathcal{Y}$, where $\mathcal{X}$ and $\mathcal{Y}$ are respectively the data and the label domains. The defining feature in this setup is that $\mathcal{X}$ is a high-dimensional space: one typically assumes $\mathcal{X} = \mathbb{R}^d$ to be a Euclidean space of large dimension $d$.

Here, it is mentioned that $N$ observations are drawn i.i.d from probability distribution $P$, which is defined over $\mathcal{X} \times \mathcal{Y}$.

My doubt is that how can we draw i.i.d from every probability distribution if our distribution is not an i.i.d distribution. The only $P$ I know to be an i.i.d is the following

$$p(x_i) = \dfrac{1}{|\mathcal{X} \times \mathcal{Y}|} \text{ for } x_i \in \mathcal{X} \times \mathcal{Y} \text{ and } 1 \le i \le |\mathcal{X} \times \mathcal{Y}|$$

To put simply, dataset with all possible $256 \times 256 \times 3$ images is i.i.d but the dataset with all dogs is not an i.i.d.

As per my knowledge, every possible distribution may not be an i.i.d distribution. Then, without knowing anything about the distribution, how can we draw i.i.d?

Answer 1

As far as I know, it doesn't make sense to say that a probability distribution is i.i.d., as you're saying.

The property i.i.d. is a property of a sequence of random variables.

In your case, the random variables are $z_i = (x_i, y_i)$, so it's not just the input $y_i$ or the label $y_i$, but both.

The rest of the explanation can be taken from the other answer that I've just given, but, in a few words, the shape of your joint doesn't determine whether your samples are i.i.d. or not.

In practice, sampling/drawing i.i.d. from a distribution could mean that you will not change the probabilities of occurrence of the samples or create other dependencies. See this article on the sampling bias.

Answer 2

My doubt is that how can we draw i.i.d from every probability distribution if our distribution is not an i.i.d distribution.

Probability distributions cannot be defined as i.i.d. or not i.i.d..

The term i.i.d. is a property of a dataset. A dataset can be created that is i.i.d. with respect to a particular probability distribution. It doesn't matter what that distribution is, it just has to exist and be relevant to the purpose the ML is being put to. An underlying population or distribution of data is assumed in a lot of theories in machine learning - for instance, it defines measures of how good a function is at deriving labels from inputs.

If you define in terms of a population, then i.i.d. is a little bit like your example $p(x_i)$ if you could sample equally likely from any member of the population. But such a population rarely consists of the product of all possible traits and all possible labels, with one example of each. Some combinations of traits and labels will be rare or impossible, and you will not expect to see them in any sample from the population.

Commonly, with real-world datasets, there is an assumption of an underlying but unknown probability distribution that is being drawn from, and some efforts are made to make the resulting dataset i.i.d. by e.g. varying free choices on the collection of data (e.g. when, where to collect), shuffling the order of samples, or by removing elements that could affect the sampling with respect to the distribution of interest (e.g. correcting for response rates by social class in surveys). This effort is worthwhile because assumptions about i.i.d. are used in practice by ML models, and they can perform worse if data is not i.i.d.

To put simply, dataset with all possible $256 \times 256 \times 3$ images is i.i.d but the dataset with all dogs is not an i.i.d.

This is incorrect. Assuming your goal is to classify dog breeds from pictures, then your first dataset is meaningless. It consists mostly of white noise images equally labeled as "poodle" or "spaniel" etc, and even when it was a recognizable picture of a dog for the $1$ in $10^{100}$ or so images where that happened, it would most likely have the wrong label. If the second dataset of dogs was all-natural photos of dogs, or all pictures of dogs found on the internet, or any other well-defined population, and it was curated properly when collected to avoid bias or correlation, then it could be i.i.d.