Skip to main content

Questions tagged [information-theory]

For questions related to information theory in the context of artificial intelligence.

8 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2 votes
0 answers
28 views

Is there any published research on the information-carrying capacity of the human face?

Is there any published research on the information-carrying capacity of the human face? Here I mean "how much information can be conveyed via facial expressions & micro-expressions". This is a ...
DukeZhou's user avatar
  • 6,237
1 vote
0 answers
90 views

Do Gradient Descent and Natural Gradient solve the same problem?

I am troubled by natural gradient methods. If we have a function f(x) we wish to minimize, gradient descent minimizes f(x) of course, but what does the natural gradient do? I found on https://...
Galois's user avatar
  • 11
1 vote
0 answers
117 views

How does NN follows law of energy conservation?

Communication requires energy, and using energy requires communication. According to Shannon, the entropy value of a piece of information provides an absolute limit on the shortest possible average ...
Satyam Kumar's user avatar
1 vote
0 answers
65 views

Loss Function In Units Of Bits?

Where can I find a machine learning library that implements loss functions measuring the Algorithmic Information Theoretic-friendly quantity "bits of ...
James Bowery's user avatar
0 votes
0 answers
10 views

NN training by optimizing hidden layer content with information bottleneck "inf_T I(X;T) - beta I(T;Y)"?

There was a lot of hype about Naftali Tishby's information bottleneck method a few years ago, but it is nearly silent now, especially that sadly the author has died in 2021. In theory it allows to ...
Jarek Duda's user avatar
0 votes
0 answers
11 views

What is the approximate minimum coding rate for NLP datasets?

I just realized that it is actually practical to use information theory to compute the maximum viable compression for datasets & that it is easiest to compute for discrete datasets. This makes me ...
profPlum's user avatar
  • 434
0 votes
0 answers
159 views

A technique to show what tokens are relatively predicted by an LLM

I’m picturing a technique where you can see what an LLM is likely to respond with, which updates in real time. It’s a bit trippy, but it’s like GitHub Copilot, in that there is predicted text while ...
Julius Hamilton's user avatar
-1 votes
1 answer
36 views

Do probabilities that don't vary much from the average contribute to greater entropy?

If the entropy of information reaches its maximum when all values of X are equally probable, indicating maximum uncertainty, do probabilities that don't vary much from the average contribute to ...
MAIKAO's user avatar
  • 1