# Tag Info

## Hot answers tagged information-theory

5

This is just an idea Given a set of pixels, the task is to decide: Which pixel is the center of an object? What is the size of the bounding boxes with the center is the pixel in part 1? Formula, consider this is a 2D image, call $(x,y)$ is the horizontal and vertical coordinate and $(w_i,h_i)$ is the size of bouding box of object $i$: $\text{For }m \in[x,x+... 3 You can know it better, if you know the concept of entropy: Information entropy is the average rate at which information is produced by a stochastic source of data. The information content (also called the surprisal) of an event${\displaystyle E}$is an increasing function of the reciprocal of the${\displaystyle p(E)}$of the event, precisely${\...

2

Information-theoretic view of Bayesian learning I once heard that the problem of approximating an unknown function can be modeled as a communication problem. How is this possible? Yes, this is indeed possible. More precisely, there is an information-theoretic view of Bayesian learning in neural networks, which can also be thought of as a communication ...

1

Apart from the entropy and the cross-entropy, which are widely used in deep learning and you seem to be aware of, there is also the Kullback-Leibler divergence (also known as relative entropy), which is widely used in the context of variational Bayesian neural networks and variational auto-encoders, given that it's often part of the loss function that is ...

1

If you have a fixed predictor, then yes. If the predictor is not fixed but deterministic, the feasibility depends on the effort needed to update the predictor, and ensuring that messages include a time stamp to ensure the correct version of the predictor is used for compression and inflation. You get a really nice property if the prediction order is in order ...

1

According to various experimentation on autoencoders, it is very possible to have latent vector of size 1. Various layers can help the downsizing of the original input to a very small size of 1. But an issue may arise during decoding. If you're expecting that through one or two or maybe five layers in decoder you can achieve an accurate reconstruction, it is ...

1

I think the wrong assumption here is that you've forgotten the cost of encoding the new features! MDL should be considered relative to the original or raw dataset. The idea is that you want to find an expression you could send to someone else that encodes the structure of the dataset in terms of the original variables. If you define new features, you need ...

Only top voted, non community-wiki answers of a minimum length are eligible