Share Your Experience: Take the 2024 Developer Survey

# Questions tagged [information-theory]

For questions related to information theory in the context of artificial intelligence.

18 questions
Filter by
Sorted by
Tagged with
36 views

### Do probabilities that don't vary much from the average contribute to greater entropy?

If the entropy of information reaches its maximum when all values of X are equally probable, indicating maximum uncertainty, do probabilities that don't vary much from the average contribute to ...
11 views

### What is the approximate minimum coding rate for NLP datasets?

I just realized that it is actually practical to use information theory to compute the maximum viable compression for datasets & that it is easiest to compute for discrete datasets. This makes me ...
• 424
1 vote
64 views

### How Mutual Information is related to uncertainty

I'm studying the chapter of Information theory from Haykin's deep learning book. It says Mutual Information between two continuous random variables $X,Y$ is defined in terms of the differential ...
• 123
333 views

### Calculating mutual information between layer outputs and targets in a neural network

I've seen in several papers that it is possible to calculate the mutual information between a layer's outputs and the desired outputs. For example: Source: https://www.ncbi.nlm.nih.gov/pmc/articles/...
• 73
140 views

### A technique to show what tokens are relatively predicted by an LLM

I’m picturing a technique where you can see what an LLM is likely to respond with, which updates in real time. It’s a bit trippy, but it’s like GitHub Copilot, in that there is predicted text while ...
85 views

### Classifier performance if data are deterministic

so let us imagine one has a classification problem at hand, say objects with $n$ numeric features, to be classified as belonging to two classes ${0,1}$. Data could look like, for $n=3$ ...
139 views

### Why would the Dice coefficient be more suitable than mutual information when you don't want 0-0 matches to be significant?

I'm confused about the interpretation and assumptions of the Dice coefficient versus the more popular measure mutual information. I'm specifically referencing its use in hierarchical semantic network ...
1 vote
90 views

I am troubled by natural gradient methods. If we have a function f(x) we wish to minimize, gradient descent minimizes f(x) of course, but what does the natural gradient do? I found on https://...
• 11
1 vote
115 views

### How does NN follows law of energy conservation?

Communication requires energy, and using energy requires communication. According to Shannon, the entropy value of a piece of information provides an absolute limit on the shortest possible average ...
335 views

### Applications of Information Theory in Machine Learning

How is information theory applied to machine learning, and in particular to deep learning, in practice? I'm more interested in concepts that yielded concrete innovations in ML, rather than theoretical ...
• 1,000
280 views

### Formal definition of the Object Detection problem

For many problems in computer science, there is a formal, mathematical problem defition. Something like: Given ..., the problem is to ... How can the Object Detection problem (i.e. detecting objects ...
• 75
1 vote
302 views

### Compressing text using AI by sending only prediction rank of next word

Is there any effort made to compress text (and maybe other media) using prediction of next word and thus sending only the order number of the word/token which will be predicted on the client side i.e ...
• 111
1 vote
65 views

### Loss Function In Units Of Bits?

Where can I find a machine learning library that implements loss functions measuring the Algorithmic Information Theoretic-friendly quantity "bits of ...
• 121
1 vote
283 views

### Is it possible to have the latent vector of an auto-encoder with size 1?

Given e.g. 1M vectors of $1000$ floating points each, where every point in vectors is sampled from a uniform distribution between $-1$ to $1$: Is it possible to have the bottleneck of the AE network ...
• 21
28 views

### Is there any published research on the information-carrying capacity of the human face?

Is there any published research on the information-carrying capacity of the human face? Here I mean "how much information can be conveyed via facial expressions & micro-expressions". This is a ...
• 6,237
1 vote
285 views

### How can a machine learning problem be reduced as a communication problem?

I once heard that the problem of approximating an unknown function can be modeled as a communication problem. How is this possible?
1 vote