Questions tagged [information-theory]

For questions related to information theory in the context of artificial intelligence.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

How to improve the PMI (Pointwise Mutual Information) Quality for document based PMI

Generating word embeddings from the PMI is well understood and known to be equivalent to SGNS (skipgram negative-sampling) under certain conditions. I was able to get good quality word embedding using ...
user avatar
  • 101
0 votes
1 answer
42 views

Why would the Dice coefficient be more suitable than mutual information when you don't want 0-0 matches to be significant?

I'm confused about the interpretation and assumptions of the Dice coefficient versus the more popular measure mutual information. I'm specifically referencing its use in hierarchical semantic network ...
user avatar
  • 1
1 vote
0 answers
39 views

Do Gradient Descent and Natural Gradient solve the same problem?

I am troubled by natural gradient methods. If we have a function f(x) we wish to minimize, gradient descent minimizes f(x) of course, but what does the natural gradient do? I found on https://...
user avatar
  • 11
1 vote
0 answers
78 views

How does NN follows law of energy conservation?

Communication requires energy, and using energy requires communication. According to Shannon, the entropy value of a piece of information provides an absolute limit on the shortest possible average ...
user avatar
1 vote
1 answer
189 views

Applications of Information Theory in Machine Learning

How is information theory applied to machine learning, and in particular to deep learning, in practice? I'm more interested in concepts that yielded concrete innovations in ML, rather than theoretical ...
user avatar
6 votes
1 answer
199 views

Formal definition of the Object Detection problem

For many problems in computer science, there is a formal, mathematical problem defition. Something like: Given ..., the problem is to ... How can the Object Detection problem (i.e. detecting objects ...
user avatar
  • 75
1 vote
1 answer
38 views

Compressing text using AI by sending only prediction rank of next word

Is there any effort made to compress text (and maybe other media) using prediction of next word and thus sending only the order number of the word/token which will be predicted on the client side i.e ...
user avatar
  • 111
1 vote
0 answers
62 views

Loss Function In Units Of Bits?

Where can I find a machine learning library that implements loss functions measuring the Algorithmic Information Theoretic-friendly quantity "bits of ...
user avatar
1 vote
1 answer
103 views

Is it possible to have the latent vector of an auto-encoder with size 1?

Given e.g. 1M vectors of $1000$ floating points each, where every point in vectors is sampled from a uniform distribution between $-1$ to $1$: Is it possible to have the bottleneck of the AE network ...
user avatar
  • 21
2 votes
0 answers
27 views

Is there any published research on the information-carrying capacity of the human face?

Is there any published research on the information-carrying capacity of the human face? Here I mean "how much information can be conveyed via facial expressions & micro-expressions". This is a ...
user avatar
  • 6,067
1 vote
1 answer
191 views

How can a machine learning problem be reduced as a communication problem?

I once heard that the problem of approximating an unknown function can be modeled as a communication problem. How is this possible?
user avatar
1 vote
1 answer
76 views

Can feature engineering change the selection of the model according to the minimum description length?

The definition of MDL according to these slides is: The minimum description length (MDL) criteria in machine learning says that the best description of the data is given by the model which ...
user avatar
2 votes
1 answer
162 views

How does the Kullback-Leibler divergence give "knowledge gained"?

I'm reading about the KL divergence on Wikipedia. I don't understand how the equation gives "information gained" as it says in the "Interpretations" section Expressed in the ...
user avatar
  • 351