# Questions tagged [entropy]

For questions about the concept of (information) entropy in the context of artificial intelligence.

4 questions
Filter by
Sorted by
Tagged with
1answer
4k views

### How to calculate the entropy in the ID3 decision tree algorithm?

Here is the definition of the entropy $$H(S)=-\sum_{x \in X} p(x) \log _{2} p(x)$$ Wikipedia's description of entropy breaks down the formula, but I still don't know how to determine the values of $X$,...
0answers
54 views

### How does NN follows law of energy conservation?

Communication requires energy, and using energy requires communication. According to Shannon, the entropy value of a piece of information provides an absolute limit on the shortest possible average ...
0answers
17 views

### Product of probabilities raised to own powers that can be used for entropy calculation

Suppose $X$ is a random variable taking $k$ values. $$Val(X) = \{x_1, x_2, x_3, \cdots, x_k\}$$ Then what is the following expression of $N(X)$ called in literature? What does it signify?  N(X) = \...
1answer
21 views

### How does high entropy targets relate to less variance of the gradient between training cases?

I've been trying to understand the Distilling the Knowledge in a Neural Network paper by Hinton et al. But I cannot fully understand this: When the soft targets have high entropy, they provide much ...