0
$\begingroup$

While studying word embeddings in natural language processing, I encountered the following statement on page 327 of the textbook Natural Language Processing by Jacob Eisenstein

Distributional semantics are computed from context statistics. Distributed semantics are a related but distinct idea: that meaning can be represented by numerical vectors rather than symbolic structures.

The dissimilarity between them is that distributed semantics represent the meaning of a word by a vector of numbers. Distributional semantics represent the meaning of a word by symbolic structure (inferred from paragraph).

I can say, in distributed semantics, the word cat can be represented by the vector $[23, 43,21,16]$ (for example).

Similarly, please, give me a small example of how the meaning of a word is represented by symbolic structure (which should not be necessarily correct).

What is meant by symbolic structure here?

$\endgroup$
1
$\begingroup$

I can't really make much sense of Eisenstein's distinction between distributional and distributed. And I think in your question you actually mix up the two terms as well, as distributed semantics involve symbolic structures, whereas distributional semantics are numerical vectors according to his definition. EDIT: actually, he seems to mix it up himself there?! Very unclear paragraph there.

I can only imagine that the symbolic structures he refers to here are semantic networks and the like, as in

(is-a feline mammal)
(is-a lion feline)
(has-a feline tail)

Here the meaning of lion, as a feline mammal with a tail, is defined through a symbolic structure, and not in reference to the context of usage. Why this should be distributed, I can only guess: the meaning components are split over a set of statements, which build up a larger structure perhaps?

It could, of course, be the case that this is covered elsewhere in the book — I haven't had the time to look through all of it.

UPDATE: Thinking more about this, perhaps he means that distributional semantics are representations where each word is a straight co-occurrence vector, ie a vector as large as the words used to define contexts, while distributed semantics is similar, but it's a different vector which is created through processing the contexts (and could thus be smaller)?

$\endgroup$
0
$\begingroup$

I am writing the answer according to my current understanding

Distributional semantics are computed from context statistics.

It is clear from the statement that the embedding of a word, in case of distributional semantics, is computed from the context statistics i.e., based on the contexts in which the word occurs.

Distributed semantics are a related but distinct idea: that meaning can be represented by numerical vectors rather than symbolic structures.

This means that embeddings obtained from distributed semantics mayn't be obtained from the symbolic structures. Now, we need to understand what is meant by symbolic structure in this case. It can be same as that of context of a word. We can understand it from the following definition of symbol structure

A physical symbol system consists of a set of entities, called symbols, which are physical patterns that can occur as components of another type of entity called an expression (symbol structure). Thus a symbol structure is composed of a number of instances (or tokens) of symbols related in some physical way (such as one token being next to another).

So, it can be understood that distributed semantics are the embeddings obtained not only through the context statistics as in the case of distributional semantics. For example, there are distributed representations beyond distributional statistics, in which embeddings are calculated from the internal structure of words and not from the context in which the word occurs (p 341). One can understand it from the following excerpt from the same page

How can word-internal structure be incorporated into word representations? One approach is to construct word representations from embeddings of the characters or morphemes.

Thus, to be concise, the embedding for the word cat is only obtained using the context statistics of cat in case of distributional semantics and in case of distributed semantics, the embedding of the word millicuries can be calculated from the embeddings the morphemes $milli,curie,s$ rather than the context statistics of the word millicuries since it is a rare word which is unlikely to have reliable context information available.

$\endgroup$
3
  • $\begingroup$ I don't think that's right. Characters don't play a role in here; I think it refers to the context of other tokens. In "the cat sat on the mat" the symbolic context would be "the", "sat", "on", "the", and "mat"; in a distributional representation these tokens would be replaced by vectors. $\endgroup$ Jun 11 at 10:40
  • $\begingroup$ @OliverMason please check now, I updated/ $\endgroup$
    – hanugm
    Jun 11 at 10:48
  • 2
    $\begingroup$ looks fine to me now. $\endgroup$ Jun 11 at 11:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.