0
$\begingroup$

I once heard that the problem of approximating an unknown function can be modeled as a communication problem. How is this possible?

$\endgroup$
0
$\begingroup$

You are probably looking for the concept of minimum description length (MDL), which is based on concepts from information theory, which was pioneered by Claude Shannon in his 1948 seminal paper A Mathematical Theory of Communication.

MDL is related to Occam's razor, which underlies many mathematical theories and frameworks, for example, AIXI, a framework for artificial general intelligence developed by Marcus Hutter. Jürgen Schmidhuber is also a good fan of Occam's razor and compression as a means to act intelligently.

The paper A Tutorial Introduction to the Minimum Description Length Principle (2004) by Peter Grünwald provides a good overview of the topic.

There are other related concepts used throughout all machine learning that are based on Shannon's entropy and information theory, for example, the KL divergence (to compute the similarity between two probability distributions) or the cross-entropy.

See also Information Theory and its Relation to Machine Learning (2015) by Bao-Gang Hu.

(By the way, Claude Shannon was one of the participants at the Dartmouth workshop, which officially started the field of artificial intelligence, so he is one of the fathers of the AI field, and his impact on the field is definitely huge).

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.