11
votes
Why exactly do neural networks require i.i.d. data?
There is an assumption behind the theory training a neural network, that also applies to many other supervised learning methods, that a training sample is representative of the data set as a whole - ...
- 32.1k
8
votes
Why exactly do neural networks require i.i.d. data?
Suppose that we have some optimization criterion $J(x)$, which we aim to optimize (maybe maximize, maybe minimize), which we can compute for a single example $x$.
In an "ideal world", where we have ...
- 10.3k
5
votes
Accepted
If the i.i.d. assumption holds, shouldn't the training and validation trends be exactly the same?
If the i.i.d (independent and identically distributed) assumption holds, shouldn't the training and validation trends be exactly the same?
No, not necessarily. Let me explain why.
If you assume your ...
- 40.5k
3
votes
How can we "draw i.i.d" from any probability distribution?
As far as I know, it doesn't make sense to say that a probability distribution is i.i.d., as you're saying.
The property i.i.d. is a property of a sequence of random variables.
In your case, the ...
- 40.5k
2
votes
Accepted
Do adversarial samples violate the i.i.d. assumption?
I believe adversarial samples violate i.i.d. by definition.
They are constructed by an adversary to undermine our model. Thus, they are not independently chosen, and are not randomly distributed.
- 2,890
2
votes
Is knowing underlying probability distribution mandatory for deciding iid property of random variables?
The point is even you know the distribution, sometimes you can't prove that the sampled data is i.i.d. or not! (more details in https://stats.stackexchange.com/q/130381/144441). Hence, without ...
- 1,816
2
votes
Accepted
Which of the following probability distribution is generating an iid dataset?
A sequence of $n$ random variables $z_{1:n} = z_1, z_2, \dots, z_n$ is i.i.d. if
they are identically distributed, i.e. each random variable $z_i$ has the same distribution
the joint distribution of ...
- 40.5k
1
vote
How can we "draw i.i.d" from any probability distribution?
My doubt is that how can we draw i.i.d from every probability distribution if our distribution is not an i.i.d distribution.
Probability distributions cannot be defined as i.i.d. or not i.i.d..
The ...
- 32.1k
1
vote
What are the iid random variables for a dataset in the GAN framework?
Independent and identically distributed random variables share the same probability distribution and each item doesn’t influence or provide insight about the value of the next item you measure. The ...
- 937
1
vote
Accepted
How would the performance of federated learning compare to the performance of centralized machine learning when the data is i.i.d.?
There are some works that do this comparison. Briefly, it's been observed that the performance of models trained via FL drops as data distributions between participating agents differ. When data is ...
- 1,000
1
vote
Are training sequences for LMs sampled in an IID fashion?
Language models explicitly assume that word sequences are not independent and identically distributed (iid). A word-based model that assumed iid within each sequence could only predict probabilities ...
- 32.1k
1
vote
Why exactly do neural networks require i.i.d. data?
Short answer
One reason why we assume/require i.i.d. data is that it simplifies the computations. More specifically, if we assume the samples to be i.i.d., their joint probability is then simplified ...
Only top scored, non community-wiki answers of a minimum length are eligible