# What does the product of probabilities raised to own powers used for entropy calculation quantify?

Suppose $$X$$ is a random variable taking $$k$$ values.

$$Val(X) = \{x_1, x_2, x_3, \cdots, x_k\}$$

Then what is the following expression of $$N(X)$$ called in literature if exists? What does it signify?

$$N(X) = \prod \limits_{i = 1}^{k} p(x_i)^{p(x_i)}$$

I am using the notation $$N(X)$$ for the sake of my convenience only.

Background: I am asking this question because of the definition of entropy I encountered. Entropy is calculated as follows.

$$H(X) = - \sum\limits_{i = 1}^{k} p(x_i) \log p(x_i)$$

If I further solve $$H(X)$$ as follows, I will get $$H(X)$$ in terms of N(X).

$$H(X) = - \sum\limits_{i = 1}^{k} p(x_i) \log p(x_i) = - \sum\limits_{i = 1}^{k} \log p(x_i)^{p(x_i)}$$ $$\implies H(X)= - \log \prod \limits_{i = 1}^{k} p(x_i)^{p(x_i)} = - \log N(X)$$

Entropy is used to characterize the unpredictability of a random variable.

A logarithm is generally applied to many quantities in AI in order to bring them into the desirable range where overflow and underflow won't happen. Hence I am thinking that $$\dfrac{1}{N(X)}$$ is the actual quantity one has to measure (the entropy?). Hence I am guessing that $$N(X)$$ can be treated as reciprocal of entropy. So, does $$N(X)$$ is a quantity that has quantified the predictability of a random variable?

$$N(X) = \dfrac{1}{2^{H(X)}} = \dfrac{1}{2^{entropy}}$$

So, I am wondering whether there is any quantity that $$N(X)$$ quantify.

• I don't know if it signifies anything or has a term, but it looks likely to underflow on a computer if you use floating point. Jun 13, 2021 at 7:52

I don't know if $$N(X)$$ has a name or has any applicability in AI, but I can comment on how this function varies as the $$H(X)$$ based on your equation

$$N(X) = \dfrac{1}{2^{H(X)}}$$

which looks correct to me (just apply the $$\log_2$$ to both sides).

In the case of a Bernoulli random variable (which is a categorical r.v. that can take 2 values, $$0$$ or $$1$$, which is a special case of your categorical r.v., if you set $$k=2$$), then this is the relationship between the probability that this random variable $$X = 1$$ and the entropy of this r.v.

So, the entropy is $$1$$ when the probability is $$0.5$$ and decreases as the probability tends to $$0$$ or $$1$$, which makes sense, because the entropy quantifies the uncertainty about the r.v. Here, the entropy is computed in bits because we use the logarithm $$\log_2$$.

So, for $$k=2$$ (in your example), then, if $$H(X) = 1$$,

$$N(X) = \dfrac{1}{2^{H(X)}} = \frac{1}{2} = \frac{1}{2}^{\frac{1}{2}} * \frac{1}{2}^{\frac{1}{2}}$$

which is equal to $$P(X = 1) = 1 - P(X = 0)$$.

Now, as $$H(X)$$ decreases to $$0$$, then $$\dfrac{1}{2^{H(X)}}$$ increases, because the denominator $$2^{H(X)}$$ becomes smaller, where the smallest value is $$H(X) = 0$$

$$N(X) = \dfrac{1}{2^{H(X)}} = 1 = 1^1 * ?$$

This is already problematic because $$0^0$$ is not well-defined. So, I already see a problem with $$N(X)$$.

For $$k > 2$$, I think the same reasoning applies, but now the maximum value of $$H(X)$$ should be $$\log_2 k$$.

So, I don't think that $$N(X)$$ is of any practical value as it can lead to expressions like $$0^0$$.

However, this type of problem also arises in the original formulation of the entropy, because $$\log_2 0$$ is not defined. The convention for when $$P(X) = 0$$ is to set $$P(X) \log P(X)$$ to zero [1].

Anyway, it seems to me that one way to interpret $$N(X)$$ is as the (average?) probability that $$X$$ takes one of the values, and it could be that the information content is what you are looking for. The definition can be found in [1].