Is there bidirection sequence-to-sequence neural machine translation?

Question

I have heard about bidirectional RNN LSTM units (endcoders-decoders), but my question is - is there bidirectional neural machine translation, that uses A->B weights for the translation in the opposite direction B->A? If not, then what are the obstacles to such system?

Answer 1

Bidirectional Mapping of Arbitrary Functions

Interchange between what properties are considered labels and what properties are considered features can support bidirectional map learning.

Unidirectional training represents the objective of functional approximation $f_p$ in terms of features $\mathcal{X}$ and labels $\mathcal{Y}$ through convergence of parameters $p$, ideally toward an accurate and reliable approximation.

$$ \mathcal{Y} \Leftarrow f_p(\mathcal{X}) $$

Because only a small proportion of arbitrary functions are monotonic (have only one value in the domain for any given value in the range), the existence of an inversion of $f_p$ is unlikely. This is certainly true of natural language processing functions. This fact will be further illuminated below.

Bidirectional training of a non-monotonic translation function $f_p$ and a reverse translation $g_p$ can be represented as two adjacent training objectives.

$$ \mathcal{Y} \Leftarrow f_p(\mathcal{X}) $$ $$ g_q(\mathcal{Y}) \Rightarrow \mathcal{X} $$

Function $g_q$ is not a guaranteed inversion of $f_p$ and may rarely be. Parameters resulting from convergence $q$ may not be derivable from parameters resulting from convergence $p$ or vice versa either.

Temporal Bidirectionality to Minimize Round Trip Inaccuracy

For reasons further illuminated below, temporal bidirectionality may be important in some use cases. For this reason, use two B-LSTM networks.

There may be some efficiency gains by training both B-LSTM networks in the same training phase, especially if more than one host, GPU, parallel signal processor, or AI accelerator is available at training time. The same is true of test and validation.

The ontology of network types is as follows.

• RNNs (recurrent artificial networks) are a stateful extension of MLPs (multi-layer perceptrons).
• LSTM (long short term memory) networks are an extension of RNNs, and use gating within each cell to adaptively regulate retention of state and its dissipation.
• B-LSTMs provide bidirectionality in the temporal direction, which allows more variety in what causality and reactance can be modeled.

Importance of Contemporary Concepts in Linguistics

Since the natural-language-processing tag was included with the posting of the question and the term, "Machine translation," was mentioned, it appears that bidirectional natural language translation using AI is the project goal. If that is the case, a contemporary linguist's understanding of words sequences in text and the sequence of phonetic elements in speech will help with developing an effective solution using AI. An engineer should be aware of the interplay between expectations placed on project results and technical feasibility at the current state of NLP technology.

• Human translation experts rely on cognition to accurately represent text that may contain cultural references, analogies, colloquialisms, and other literary or oratory devices.

• Because of rhyme, alliteration, cadence and other phonetic devices, prose and poetry may not have an equivalent in the target language.

• Linguistic elements are not strictly words. A single word may have multiple linguistic elements because of tenses, plurality, gender, prefixes, suffixes, and other modifiers. For example, "Multicolored," is an adjective made up of three linguistic elements: {"multi-", "color", "-ed"}. A single idea is sometimes represented as multiple words, the individual meanings and juxtaposition of which does not imply the meaning of the word group. For example, "I don't wish to hitch my apple-cart to that wobbly wagon," has nothing to do with apples, carts, mechanical stability, or wagons.

• The translation $\mathcal{A} \Rightarrow \mathcal{B} \Rightarrow \mathcal{C}$ may not lead to $\mathcal{A} = \mathcal{C}$.

• When a series of linguistic elements are communicated, the semantics represented are not often a series. For instance, "Flying birds can also jump," is equivalent to, "If a bird is able to fly, it can jump." This works in natural language because the associations and modifying aspects between linguistic elements is a semantic network that correlates with a cognitive ontology. The serialization of this network and its reconstitution in another mind cannot be formalized in a way that eliminates all ambiguity. Communication $\mathcal{K}_o \Rightarrow \mathcal{L} \Rightarrow \mathcal{K}_r$, where $\mathcal{K}$ are cognitions at originating intelligence $o$ and target intelligence $t$ respectively and $\mathcal{L}$ is a language expression in between, may not lead to $\mathcal{K}_t = \mathcal{K}_o$.

• For the above reasons, formal grammar has been discarded as a set of principles that can facilitate NLP. Humans neither write nor speak in conformance with any formal grammar on a regular basis.

Expected Dominance of Recursive Network Technology

For this reason, LSTM networks may not be the direction of NLP technology development. Recursive network designs (often using RNN as an acronym but referring to a distinct subset of recurrent networks) can train semantic network models based on serial streams of linguistic elements. That appears to bear closest resemblance to language centers in the human mind, and may emerge as a dominant NLP component in emerging AI systems.