Skip to main content

Questions tagged [information-theory]

For questions related to information theory in the context of artificial intelligence.

Filter by
Sorted by
Tagged with
6 votes
1 answer
282 views

Formal definition of the Object Detection problem

For many problems in computer science, there is a formal, mathematical problem defition. Something like: Given ..., the problem is to ... How can the Object Detection problem (i.e. detecting objects ...
JavAlex's user avatar
  • 75
3 votes
1 answer
365 views

Calculating mutual information between layer outputs and targets in a neural network

I've seen in several papers that it is possible to calculate the mutual information between a layer's outputs and the desired outputs. For example: Source: https://www.ncbi.nlm.nih.gov/pmc/articles/...
VJ123's user avatar
  • 83
2 votes
1 answer
385 views

How does the Kullback-Leibler divergence give "knowledge gained"?

I'm reading about the KL divergence on Wikipedia. I don't understand how the equation gives "information gained" as it says in the "Interpretations" section Expressed in the ...
Gooby's user avatar
  • 351
2 votes
1 answer
340 views

Applications of Information Theory in Machine Learning

How is information theory applied to machine learning, and in particular to deep learning, in practice? I'm more interested in concepts that yielded concrete innovations in ML, rather than theoretical ...
SpiderRico's user avatar
  • 1,000
2 votes
0 answers
28 views

Is there any published research on the information-carrying capacity of the human face?

Is there any published research on the information-carrying capacity of the human face? Here I mean "how much information can be conveyed via facial expressions & micro-expressions". This is a ...
DukeZhou's user avatar
  • 6,237
1 vote
1 answer
294 views

How can a machine learning problem be reduced as a communication problem?

I once heard that the problem of approximating an unknown function can be modeled as a communication problem. How is this possible?
Raphael Augusto's user avatar
1 vote
1 answer
73 views

How Mutual Information is related to uncertainty

I'm studying the chapter of Information theory from Haykin's deep learning book. It says Mutual Information between two continuous random variables $X,Y$ is defined in terms of the differential ...
piero's user avatar
  • 123
1 vote
1 answer
304 views

Compressing text using AI by sending only prediction rank of next word

Is there any effort made to compress text (and maybe other media) using prediction of next word and thus sending only the order number of the word/token which will be predicted on the client side i.e ...
sktguha's user avatar
  • 111
1 vote
1 answer
287 views

Is it possible to have the latent vector of an auto-encoder with size 1?

Given e.g. 1M vectors of $1000$ floating points each, where every point in vectors is sampled from a uniform distribution between $-1$ to $1$: Is it possible to have the bottleneck of the AE network ...
ENECO's user avatar
  • 21
1 vote
1 answer
126 views

Can feature engineering change the selection of the model according to the minimum description length?

The definition of MDL according to these slides is: The minimum description length (MDL) criteria in machine learning says that the best description of the data is given by the model which ...
user avatar
1 vote
0 answers
90 views

Do Gradient Descent and Natural Gradient solve the same problem?

I am troubled by natural gradient methods. If we have a function f(x) we wish to minimize, gradient descent minimizes f(x) of course, but what does the natural gradient do? I found on https://...
Galois's user avatar
  • 11
1 vote
0 answers
117 views

How does NN follows law of energy conservation?

Communication requires energy, and using energy requires communication. According to Shannon, the entropy value of a piece of information provides an absolute limit on the shortest possible average ...
Satyam Kumar's user avatar
1 vote
0 answers
65 views

Loss Function In Units Of Bits?

Where can I find a machine learning library that implements loss functions measuring the Algorithmic Information Theoretic-friendly quantity "bits of ...
James Bowery's user avatar
0 votes
1 answer
87 views

Classifier performance if data are deterministic

so let us imagine one has a classification problem at hand, say objects with $n$ numeric features, to be classified as belonging to two classes ${0,1}$. Data could look like, for $n=3$ ...
Smerdjakov's user avatar
0 votes
1 answer
142 views

Why would the Dice coefficient be more suitable than mutual information when you don't want 0-0 matches to be significant?

I'm confused about the interpretation and assumptions of the Dice coefficient versus the more popular measure mutual information. I'm specifically referencing its use in hierarchical semantic network ...
Arden's user avatar
  • 1
0 votes
0 answers
10 views

NN training by optimizing hidden layer content with information bottleneck "inf_T I(X;T) - beta I(T;Y)"?

There was a lot of hype about Naftali Tishby's information bottleneck method a few years ago, but it is nearly silent now, especially that sadly the author has died in 2021. In theory it allows to ...
Jarek Duda's user avatar
0 votes
0 answers
11 views

What is the approximate minimum coding rate for NLP datasets?

I just realized that it is actually practical to use information theory to compute the maximum viable compression for datasets & that it is easiest to compute for discrete datasets. This makes me ...
profPlum's user avatar
  • 434
0 votes
0 answers
159 views

A technique to show what tokens are relatively predicted by an LLM

I’m picturing a technique where you can see what an LLM is likely to respond with, which updates in real time. It’s a bit trippy, but it’s like GitHub Copilot, in that there is predicted text while ...
Julius Hamilton's user avatar
-1 votes
1 answer
36 views

Do probabilities that don't vary much from the average contribute to greater entropy?

If the entropy of information reaches its maximum when all values of X are equally probable, indicating maximum uncertainty, do probabilities that don't vary much from the average contribute to ...
MAIKAO's user avatar
  • 1