# Questions tagged [entropy]

For questions about the concept of (information) entropy in the context of artificial intelligence.

8 questions
Filter by
Sorted by
Tagged with
5k views

### How to calculate the entropy in the ID3 decision tree algorithm?

Here is the definition of the entropy $$H(S)=-\sum_{x \in X} p(x) \log _{2} p(x)$$ Wikipedia's description of entropy breaks down the formula, but I still don't know how to determine the values of $X$,...
• 131
1 vote
30 views

### Can entropy bonus be used with state-independent log std for stochastic policies?

In this blog article by openai, they say the std of the exploration distribution must be state-dependent, i.e. an output of the policy network, so it works with the entropy bonus, which is an integral ...
• 131
1 vote
21 views

### Why exclude the first entropy bonus term in the soft Q-function in SAC?

Based on OpenAI Spinning Up description of Soft Actor Critic (SAC) the soft Q-function is defined as and as they say Q value is changed to include the entropy bonuses from every timestep except the ...
• 11
1 vote
86 views

### How does NN follows law of energy conservation?

Communication requires energy, and using energy requires communication. According to Shannon, the entropy value of a piece of information provides an absolute limit on the shortest possible average ...
54 views

### What is the relationship between entropy in thermodynamics and entropy in information theory?

BACKGROUND: In thermodynamics, entropy $S$ is a measure of disorder and is given by $${\displaystyle S=k_B\log(W)},$$ where $k_B$ is Boltzman's constant and $W$ is the number of microstates. In ...
• 767
57 views

### What does the product of probabilities raised to own powers used for entropy calculation quantify?

Suppose $X$ is a random variable taking $k$ values. $$Val(X) = \{x_1, x_2, x_3, \cdots, x_k\}$$ Then what is the following expression of $N(X)$ called in literature if exists? What does it signify? ...
• 3,391