Usually, our training objective over a batch is written in terms of the expected value of a sample in that batch such as

$objective = E_{x \sim data} * log(P(x))$

But in the code implementations, these objectives are averaged over the batch, even if it very obvious that they might not have been sampled uniformly.

Is this just for simplification purposes or is there backing to both of them being equivalent in the training sense?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.