# How can we "draw i.i.d" from any probability distribution?

Consider the following paragraph from 2 Learning in High Dimensions in from of the paper titled Geometric Deep Learning Grids, Groups, Graphs, Geodesics, and Gauges

Supervised machine learning, in its simplest formalisation, considers a set of $$N$$ observations $$D = \{(x_i, y_i)\}_{i=1}^{N}$$ drawn i.i.d. from an underlying data distribution $$P$$ defined over $$\mathcal{X} \times \mathcal{Y}$$, where $$\mathcal{X}$$ and $$\mathcal{Y}$$ are respectively the data and the label domains. The defining feature in this setup is that $$\mathcal{X}$$ is a high-dimensional space: one typically assumes $$\mathcal{X} = \mathbb{R}^d$$ to be a Euclidean space of large dimension $$d$$.

Here, it is mentioned that $$N$$ observations are drawn i.i.d from probability distribution $$P$$, which is defined over $$\mathcal{X} \times \mathcal{Y}$$.

My doubt is that how can we draw i.i.d from every probability distribution if our distribution is not an i.i.d distribution. The only $$P$$ i know to be an i.i.d is the following

$$p(x_i) = \dfrac{1}{|\mathcal{X} \times \mathcal{Y}|} \text{ for } x_i \in \mathcal{X} \times \mathcal{Y} \text{ and } 1 \le i \le |\mathcal{X} \times \mathcal{Y}|$$

To put simply, dataset with all possible $$256 \times 256 \times 3$$ images is i.i.d but the dataset with all dogs is not an i.i.d.

As per my knowledge, every possible distribution mayn't be an i.i.d distribution. Then, without knowing anything about the distribution, how can we draw i.i.d?

• What do you mean by "our distribution is not an i.i.d distribution"? i.i.d. means independent and identically distributed, and it's something associated with a random variable in relation to other random variables. It may be a good idea to read something about the topic before asking a question, otherwise, there's the risk of making wrong assumptions. So, as far as I know, it doesn't make any sense to say "a prob. distribution is not i.i.d.".
– nbro
Sep 11 at 12:24
• @nbro I am aware of the fact that iid applies to random variables, you can check my previous question on it. But, inorder to verify whether random variables are iid or not, we need to check with probability distribution only right? Sep 11 at 12:28
• @nbro I will read this answer once and check where I am going wrong. Sep 11 at 12:31
• "a prob. distribution is not i.i.d." I mean rv's included in the distribution. @nbro Sep 11 at 12:38
• Ok (yes, a r.v. has an associated prob. distribution, as I just said!), but this does not imply that you can say "rv's included in the distribution". I'm just trying to fix the terminology you're using because it seems that you're misunderstanding some concepts. So, I would start by changing "distribution is not an i.i.d distribution" to something that makes sense, although Neil already tried to address this misconception.
– nbro
Sep 11 at 12:46

My doubt is that how can we draw i.i.d from every probability distribution if our distribution is not an i.i.d distribution.

Probability distributions cannot be defined as i.i.d. or not i.i.d..

The term i.i.d. is a property of a dataset. A dataset can be created that is i.i.d. with respect to a particular probability distribution. It doesn't matter what that distribution is, it just has to exist, and be relevant to the purpose the ML is being put to. An underlying population or distribution of data is assumed in a lot of theory in machine learning - for instance it defines measures of how good a function is at deriving labels from inputs.

If you define in terms of a population, then i.i.d. is a little bit like your example $$p(x_i)$$ if you could sample equally likely from any member of the population. But such a population rarely consists of the product of all possible traits and all possible labels, with one example of each. Some combinations of traits and labels will be rare or impossible, and you will not expect to see them in any sample from the population.

Commonly, with real world datasets, there is an assumption of an underlying but unknown probability distribution that is being drawn from, and some efforts are made to make the resulting dataset i.i.d. by e.g. varying free choices on collection of data (e.g. when, where to collect), shuffling the order of samples, or by removing elements that could affect the sampling with respect to the distribution of interest (e.g. correcting for response rates by social class in surveys). This effort is worthwhile because assumptions about i.i.d. are used in practice by ML models, and they can perform worse if data is not i.i.d.

To put simply, dataset with all possible $$256 \times 256 \times 3$$ images is i.i.d but the dataset with all dogs is not an i.i.d.

This is incorrect. Assuming your goal is to classify dog breeeds from pictures, then your first dataset is meaningless. It consists mostly of white noise images equally labelled as "poodle" or "spaniel" etc, and even when it was a recognisable picture of a dog for the $$1$$ in $$10^{100}$$ or so images where that happened, it would most likely have the wrong label. If the second dataset of dogs was all natural photos of dogs, or all pictures of dogs found on the internet, or any other well-defined population, and it was curated properly when collected to avoid bias or correlation, then it could be i.i.d.