The capsule neural network seems to be a good solution for problems that involve hierarchies. For example, a face is composed of eyes, a nose and ears; a hand is made of fingers, nails, and a palm; and a human is composed of a face and hands.

Many problems in NLP can be seen as hierarchical problems: there are words, sentences, paragraphs, and chapters, whose meaning changes based on the style of lower levels.

Are there any research papers (which I should be aware of) on the application of capsule neural networks to NLP problems?

Are there related research papers, which have been investigating hierarchical complexity within the domain of NLP, which could be easily translated to Capsule Network?

  • 1
    $\begingroup$ I think there might be something like NNP "Natural Narrative Processing", which would be the superset of NLP. $\endgroup$ – Ahti Ahde Dec 18 '17 at 11:55

For us to answer this question. First, we need to look at why capsule networks outperform convolution neural networks by as much as 45% in recognizing images that have been rotated, translated or are under a different pose. We can find Geof Hinton's paper on capsule networks here for reference https://arxiv.org/pdf/1710.09829v1.pdf

In a CNN architecture, a convolution layer is usually followed by a max-pooling layer. This is so that the lower levels can detect low-level features, like edges, while the high-level layers can detect abstraction like eyes. However, the application of max-pooling leads to the loss of important information regarding the location and spatial relationship between certain features.

On the other hand, this is where capsule networks excel, the way they represent certain features is locally invariant. This is why capsule networks can recognize images under different lighting conditions and deformations. They are likely to excel at applications such as video and object tracking but not necessarily NLP.

The current approach in NLP maps words and phrases to vectors. From there, we exploit the concept of vectors and distances between them (cosine, euclidean, etc.) to perform operations such as: finding the similarity between words and even documents, machine translation, and natural language understanding (NLU).

Capsule networks are unlikely to succeed in NLP. This is because algorithms that aim to find the hierarchical structure of natural languages or approaches that focus on grammar have met little success. Research by Stanford University aiming at finding the hierarchical structure of natural languages can be found here https://nlp.stanford.edu/projects/project-induction.shtml

Although conclusive research regarding other applications of capsule networks has not yet been conducted. They are likely to excel at applications such as video intelligence and object tracking but not necessarily NLP.

  • $\begingroup$ It seems that it no longer is true that "Capsule Networks are unlikely to succeed in NLP" as Hinton himself is conducting research on this at Google; thus I think I should uncorrect this answer. I do understand that conceptually it is hard to distinct where NLP ends and something else starts, however, I would believe Thought Vectors for example could provide a similar addition to NLP as Dependency Grammars; they could detect the difference of thematic meaningness of words far better than n-gram based methods. $\endgroup$ – Ahti Ahde Jun 15 '20 at 14:39

Geoffrey Hinton has started working on Thought Vectors at Google: https://en.wikipedia.org/wiki/Thought_vector

The basic idea is similar to his original idea with Capsule Networks, where activation happens by vectors instead of scalars, which allows the network to capture transformations: for example while traditional CNN needs to see object from all perspectives of three dimensional space, the Capsule networks are able to extrapolate transformations such as stretching much better.

Thought Vectors guide NLP similarly; one could say that there are two grammars, the linguistic grammar and the narrative grammar which is more universal (Vladimir Propp, Joseph Campbell, John Vervake). While dependency grammars do great job at understanding linguistic grammar, we lack tools for meaning extraction, which is narrative bound. Thus Thought Vectors could, at least in theory, give us a framework for matching the meaning of a word within a context rather than just lexically and grammarly trying to approximate the meaning through average co-occurances.

Neural Networks with Thought Vectors would be highly complex and beyond our computational resources today (Hinton predicts in one paper, that we would get there around 2035), however, one could conduct empirical research already by giving a heuristic structure for Thought Vectors by utilizing narrative systems that do compute more easily. One could for example have text segments annotated with writing theories or other such devices that would approximate the Thought Vectors conceptually. For example annotating the text with state transformations of conflict driven partially ordered causal link planner (cPOCL, Gervas et al.) or use a writing theory framework such as Dramatica to annotate known movie scripts (http://dramatica.com/theory http://dramatica.com/analysis).

Hinton himself is currently active in NLP research: https://research.google/people/GeoffreyHinton/

Here is a nice explanation of Thought Vectors: https://pathmind.com/wiki/thought-vectors


There has been some recent work on this: Investigating Capsule Networks with Dynamic Routing for Text Classification

Seems some are having some success with it.

  • 3
    $\begingroup$ Instead of just a link, it would be more useful if you could list the title of the paper and what it is about, so that people can assess the relevance before downloading it. $\endgroup$ – Oliver Mason Jun 25 '18 at 8:20
  • $\begingroup$ Well, it would be about Capsule networks and NLP. That doesn't seem to be to great a conceptual leap for most people to make. $\endgroup$ – user2202800 Jun 26 '18 at 16:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.