Suppose $X$ is a random variable taking $k$ values.

$$Val(X) = \{x_1, x_2, x_3, \cdots, x_k\} $$

Then what is the following expression of $N(X)$ called in literature? What does it signify?

$$ N(X) = \prod \limits_{i = 1}^{k} p(x_i)^{p(x_i)}$$

I am using the notation $N(X)$ for the sake of my convenience only.

Background: I am asking this question because of the definition of entropy I encountered. Entropy is calculated as follows.

$$ H(X) = - \sum\limits_{i = 1}^{k} p(x_i) \log p(x_i) $$

If I further solve $H(X)$ as follows, I will get $H(X)$ interms of N(X).

$$ H(X) = - \sum\limits_{i = 1}^{k} p(x_i) \log p(x_i) = - \sum\limits_{i = 1}^{k} \log p(x_i)^{p(x_i)} $$ $$\implies H(X)= - \log \prod \limits_{i = 1}^{k} p(x_i)^{p(x_i)} = - \log N(X)$$

Entropy is used to characterize the unpredictability of a random variable.

  • 2
    $\begingroup$ I don't know if it signifies anything or has a term, but it looks likely to underflow on a computer if you use floating point. $\endgroup$ Jun 13 at 7:52
  • $\begingroup$ Maybe I missed it, but that paper (you're linking us to) does not show the derivation you're showing us. So, can you clarify where you found this derivation? In any case, that derivation seems to come from the properties of logarithms. $\endgroup$
    – nbro
    Jun 14 at 1:56
  • $\begingroup$ @nbro I did my own derivation based on properties of logarithm. The linked pdf contains only definition. $\endgroup$
    – hanugm
    Jun 14 at 2:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.