Questions tagged [information-theory]

For questions related to information theory in the context of artificial intelligence.

Filter by
Sorted by
Tagged with
0
votes
0answers
23 views

Do Gradient Descent and Natural Gradient solve the same problem?

I am troubled by natural gradient methods. If we have a function f(x) we wish to minimize, gradient descent minimizes f(x) of course, but what does the natural gradient do? I found on https://...
0
votes
0answers
11 views

Controlling mutual information in latent variables

Recently, I read some interesting papers on mutual information (MI) estimation in high dimensional variables using neural networks [Belghazi et al., 2018][Poole et al., 2019]. These methods besides ...
1
vote
0answers
54 views

How does NN follows law of energy conservation?

Communication requires energy, and using energy requires communication. According to Shannon, the entropy value of a piece of information provides an absolute limit on the shortest possible average ...
1
vote
1answer
116 views

Applications of Information Theory in Machine Learning

How is information theory applied to machine learning, and in particular to deep learning, in practice? I'm more interested in concepts that yielded concrete innovations in ML, rather than theoretical ...
6
votes
1answer
137 views

Formal definition of the Object Detection problem

For many problems in computer science, there is a formal, mathematical problem defition. Something like: Given ..., the problem is to ... How can the Object Detection problem (i.e. detecting objects ...
1
vote
1answer
29 views

Compressing text using AI by sending only prediction rank of next word

Is there any effort made to compress text (and maybe other media) using prediction of next word and thus sending only the order number of the word/token which will be predicted on the client side i.e ...
1
vote
0answers
57 views

Loss Function In Units Of Bits?

Where can I find a machine learning library that implements loss functions measuring the Algorithmic Information Theoretic-friendly quantity "bits of ...
1
vote
1answer
46 views

Is it possible to have the latent vector of an auto-encoder with size 1?

Given e.g. 1M vectors of $1000$ floating points each, where every point in vectors is sampled from a uniform distribution between $-1$ to $1$: Is it possible to have the bottleneck of the AE network ...
2
votes
0answers
27 views

Is there any published research on the information-carrying capacity of the human face?

Is there any published research on the information-carrying capacity of the human face? Here I mean "how much information can be conveyed via facial expressions & micro-expressions". This is a ...
1
vote
1answer
151 views

How can a machine learning problem be reduced as a communication problem?

I once heard that the problem of approximating an unknown function can be modeled as a communication problem. How is this possible?
1
vote
1answer
55 views

Can feature engineering change the selection of the model according to the minimum description length?

The definition of MDL according to these slides is: The minimum description length (MDL) criteria in machine learning says that the best description of the data is given by the model which ...
2
votes
1answer
77 views

How does the Kullback-Leibler divergence give “knowledge gained”?

I'm reading about the KL divergence on Wikipedia. I don't understand how the equation gives "information gained" as it says in the "Interpretations" section Expressed in the ...