3
$\begingroup$

The specific problem I'm having is with a Fully Visible Belief Network. It is an explicit density model (though I don't know what quantifies something being such) that uses the chain rule to decompose the likelihood of an image x into a product of 1-d distributions.

enter image description here

What is meant by "the likelihood of an image x"? With respect to what? I assume it refers to how common this image would be in the data set it is selected from? Like if you had 1000 images, 800 of which were white and 200 of which were black, the model should ideally output 0.2 for any black image inputted? Obliviously with more complicated clustering like dogs vs cats it'd be a bit different, but that's my intuition. Is that correct?

Also as a side question, that equation looks very wrong. If you have an image of $1048\times720$ pixels, and say every pixel evaluates to have a probability of 0.9, you'd expect the final probability of the image to be 0.9 or 90%. But according to that equation, it's $0.9^{720*1048}$, which is stupidly small, essentially 0. What's going on here?

Improve this question
$\endgroup$
4
  • $\begingroup$ After normalization it wouldn't look stupidly small. Read 'Deep Learning' by Goodfellow (undirected graphical models) and also some YouTube lectures by Ali Ghodsi on RBMs. I also had somewhat of a similar problem in understanding these concepts. $\endgroup$
    – user9947
    Dec 6, 2019 at 12:11
  • $\begingroup$ Sorry wrong statement, the correct would be that compared to other probablity it will be still quite large. $\endgroup$
    – user9947
    Dec 6, 2019 at 12:19
  • $\begingroup$ @DuttaA Ok that makes sense. Do you know if my first assumption about the meaning of "likelihood of an image x" is correct? $\endgroup$
    – Recessive
    Dec 6, 2019 at 13:47
  • 2
    $\begingroup$ In practice, you often work with log probabilities, rather than probabilities. Maybe later I will have a closer look at this question and maybe try to provide an answer. $\endgroup$
    – nbro
    Dec 6, 2019 at 17:16

1 Answer 1

Reset to default
2
$\begingroup$

When you say likelihood, you are invoking several other concepts like events, sample, parameters, model, probability density function (PDF), etc (it would be helpful if you learn more about these concepts). In essence, a likelihood function $l(x|\theta)$ is a PDF that quantifies how likely is that event $x$ happens out of a set of possible events, given the parameters $\theta$ that define your model.

In the specific case of images, the set of possible events is usually one of two options: 1) all the available images in a dataset, or 2) all the existing images. Usually you want to model the likelihood in the option 2), but only having access to a sample of all the possible images. In either case, the likelihood is just the probability that you select one image out of all the possible ones. If you consider only images of $1048\times 720$ pixels, the possible amount of images is $(256\times3)^{1048\times 720}$, where I am assuming that each pixel consists of 3 colors and each color can take 256 values. Since the amount of possible images is so so big, it is very common that the probability of selecting a specific one is very very small. This is a reason why you usually work with log-likelihood (the logarithm of the likelihood) instead of directly using likelihoods. For example, if all your images were equally probable, the likelihood would be in the order of $10^{-{10^7}}$, while the log-likelihood would be around $-10^7$.

To solve your paradox with the probability of images and pixels, consider that instead of pixels you have coins and instead of images you have sequences of coins. Let's say that you have a fair coin, so the probability of tails ($T$) after tossing the coin is 0.5. If you toss a second coin, the probability of having $T$ again is naturally 0.5 as well, but what is the probability that both results where $T$? It is the product (0.25) since the events are independent. Similarly, the probability of the other three sequences $TH$, $HT$ and $HH$ is just 0.25. You can see that since the probability needs to be shared between 4 sequences equally probable, they are less probable with respect to the probabilities of the sequences of length 1. If you toss the coin 3 times, then the probability of all these coins being tails is just $0.5^{3}$. Again, there are now 8 possible sequences, all sharing the same amount of probability. You can see what's going on. Since the amount of possible options becomes large, the probability of each possible sequence of coins becomes small. Clearly, you would never toss a coin 10 times and expect to obtain all $T$, right? Well, exactly the same happens in the case of the pictures.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the answer! It makes sense why you would multiply, and use a log. I'm still a bit confused as to why a PDF in this case is being used for a discrete set (while it is massive, the set of all images of defined dimensions is finite). Moreover, normally when you want to find the probability of an event you don't find the exact events probability, but a set of events with some tolerance as an estimate, as an exact event occurring has a likelihood of 0. Is that done in unsupervised learning? $\endgroup$
    – Recessive
    Dec 7, 2019 at 1:51
  • 1
    $\begingroup$ @Recessive, you are right in saying that the likelihood is a prob. mass function if the set of events is discrete. It just seemed more general to refer to a density function. Now, in the case of images, actually it is more common to suppose that the pixel values are continuous and that the likelihood is a PDF. This supposition is made precisely because the set of events is so massive and the probabilities become so insignificant. The numbers I used were mostly to try to give you an intuition. $\endgroup$
    – Diego Gomez
    Dec 8, 2019 at 21:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .