0
$\begingroup$

We should

conceptualize LLMs as very quirky and very experimental if advanced information retrieval systems (fuzzy JPEGs)

according to point 7 in the screenshot below (grabbed from pmarca's Twitter feed).

Could someone please help me understand how "fuzzy" and "JPEGs" are relevant here?

enter image description here

$\endgroup$
6
  • $\begingroup$ JPEGs are not a great analogy, but the idea is lossy compression. If you use arithmetic coding on the indices of correct output from the LLM, then minimizing cross-entropy loss minimizes your coding scheme. If you were to always use the most likely output (or only sample from the top 10% of tokens), you'd have a lossy compression. $\endgroup$ Apr 1 at 4:35
  • $\begingroup$ @programjames here, does "fuzzy" mean "fuzzy logic" or a "not sharp image"? $\endgroup$ Apr 1 at 6:09
  • $\begingroup$ The latter. JPEGs are already lossy, so it wouldn't make sense to add more noise when reconstructing them. The hidden layers of LLMs do happen to implement fuzzy logic though, so that is confusing. $\endgroup$ Apr 2 at 0:33
  • $\begingroup$ For reproducibility, please, provide the link to the tweet $\endgroup$
    – nbro
    Apr 2 at 14:31
  • 1
    $\begingroup$ @nbro done. twitter.com/pmarca/status/1774547180713631885 $\endgroup$ Apr 2 at 14:48

1 Answer 1

0
$\begingroup$

As a psychologist, Alison Gopnik's analogy of a "fuzzy jpeg" refers to human brain development. Young children's brains might be like a "fuzzy jpeg," but they can still demonstrate reasonable learning and knowledge. She extended the analogy to artificial intelligence/machine learning. While a neural network is still fairly "fuzzy," it can still produce reasonable and sometimes surprising results.

Of course, at the risk of overfitting, the more we train a neural network the less fuzzy it becomes.

But I think it is a poor analogy, given that a JPEG is a static image and a neural network is proven to be Turing complete (computationally universal).

$\endgroup$
3
  • $\begingroup$ I think some mixup on "Turing complete". The universal approximation theorem says nothing about that. I believe that recurrent neural networks can be viewed as trainable Turing machines, but not sure if there's proof of universal computation? If you have a reference for the proof you mention at the end, I would appreciate a link $\endgroup$ Apr 3 at 11:47
  • 1
    $\begingroup$ The Universal approximation theorem only addresses continuous functions, not sufficient to be Turing complete. I have several references to papers that address neural networks as Turing complete, therefore also computational universal. I have a list of attributions to papers that prove certain classes and configurations of ANNs as Turing complete. I will share the links to the online versions I can find. $\endgroup$ Apr 3 at 19:41
  • 1
    $\begingroup$ Here are a few relevant papers addressing the topic of Turing completeness of ANNs, there are many more. "How Powerful are Decoder-Only Transformer Neural Models?" arxiv.org/pdf/2305.17026.pdf "On the Turing Completeness of Modern Neural Network Architecture" arxiv.org/abs/1901.03429 "Attention is Turing Complete" jmlr.org/papers/volume22/20-302/20-302.pdf $\endgroup$ Apr 3 at 19:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .