Questions tagged [information-theory]
For questions related to information theory in the context of artificial intelligence.
18
questions
0
votes
1
answer
23
views
How Mutual Information is related to uncertainty
I'm studying the chapter of Information theory from Haykin's deep learning book.
It says Mutual Information between two continuous random variables $X,Y$ is defined in terms of the differential ...
- 103
2
votes
1
answer
152
views
Calculating mutual information between layer outputs and targets in a neural network
I've seen in several papers that it is possible to calculate the mutual information between a layer's outputs and the desired outputs. For example:
Source: https://www.ncbi.nlm.nih.gov/pmc/articles/...
- 73
0
votes
0
answers
100
views
A technique to show what tokens are relatively predicted by an LLM
I’m picturing a technique where you can see what an LLM is likely to respond with, which updates in real time.
It’s a bit trippy, but it’s like GitHub Copilot, in that there is predicted text while ...
- 133
1
vote
0
answers
17
views
Can a language model detect that a concept assumed by humanity as true is false? [duplicate]
With the arrival of AI-based search engines, I wonder if these models, being trained on human texts, always offer the solution most accepted by most of humans.
This raises the question of whether ...
user69389
0
votes
0
answers
24
views
Is there a most general-purpose unsupervised learning algorithm?
I was thinking about training a model on non-linguistic material like video, and I was wondering if it could form concepts about the world, and also somehow form composite concepts or conceptual ...
- 133
0
votes
1
answer
83
views
Classifier performance if data are deterministic
so let us imagine one has a classification problem at hand, say objects with $n$ numeric features, to be classified as belonging to two classes ${0,1}$.
Data could look like, for $n=3$
...
- 45
0
votes
1
answer
111
views
Why would the Dice coefficient be more suitable than mutual information when you don't want 0-0 matches to be significant?
I'm confused about the interpretation and assumptions of the Dice coefficient versus the more popular measure mutual information. I'm specifically referencing its use in hierarchical semantic network ...
- 1
1
vote
0
answers
81
views
Do Gradient Descent and Natural Gradient solve the same problem?
I am troubled by natural gradient methods.
If we have a function f(x) we wish to minimize, gradient descent minimizes f(x) of course, but what does the natural gradient do?
I found on https://...
- 11
1
vote
0
answers
110
views
How does NN follows law of energy conservation?
Communication requires energy, and using energy requires communication. According to Shannon, the entropy value of a piece of information provides an absolute limit on the shortest possible average ...
- 67
2
votes
1
answer
302
views
Applications of Information Theory in Machine Learning
How is information theory applied to machine learning, and in particular to deep learning, in practice? I'm more interested in concepts that yielded concrete innovations in ML, rather than theoretical ...
- 980
6
votes
1
answer
252
views
Formal definition of the Object Detection problem
For many problems in computer science, there is a formal, mathematical problem defition.
Something like: Given ..., the problem is to ...
How can the Object Detection problem (i.e. detecting objects ...
- 75
1
vote
1
answer
276
views
Compressing text using AI by sending only prediction rank of next word
Is there any effort made to compress text (and maybe other media) using prediction of next word and thus sending only the order number of the word/token which will be predicted on the client side
i.e
...
- 111
1
vote
0
answers
65
views
Loss Function In Units Of Bits?
Where can I find a machine learning library that implements loss functions measuring the Algorithmic Information Theoretic-friendly quantity "bits of ...
- 121
1
vote
1
answer
264
views
Is it possible to have the latent vector of an auto-encoder with size 1?
Given e.g. 1M vectors of $1000$ floating points each, where every point in vectors is sampled from a uniform distribution between $-1$ to $1$:
Is it possible to have the bottleneck of the AE network ...
- 21
2
votes
0
answers
28
views
Is there any published research on the information-carrying capacity of the human face?
Is there any published research on the information-carrying capacity of the human face?
Here I mean "how much information can be conveyed via facial expressions & micro-expressions".
This is a ...
- 6,237
1
vote
1
answer
254
views
How can a machine learning problem be reduced as a communication problem?
I once heard that the problem of approximating an unknown function can be modeled as a communication problem. How is this possible?
- 187
1
vote
1
answer
117
views
Can feature engineering change the selection of the model according to the minimum description length?
The definition of MDL according to these slides is:
The minimum description length (MDL) criteria in machine learning says that the best description of the data is given by the model which ...
user9947
2
votes
1
answer
353
views
How does the Kullback-Leibler divergence give "knowledge gained"?
I'm reading about the KL divergence on Wikipedia. I don't understand how the equation gives "information gained" as it says in the "Interpretations" section
Expressed in the ...
- 351