Tag Info

5

This is my own understanding of hidden state in a recurrent network and if its wrong please feel free to let me know. Lets take this simple sequence first, X = [a,b,c,d,.......,y,z] Y = [b,c,d,e,.......,z,a] Instead of RNN we will first try to train this in a simple multi layer neural network with one input and one output, here hidden layers details ...

4

Usually, in natural language processing (NLP), they are using Sequence to Sequence Learning (Seq2Seq) with Neural Networks, such as Recurrent Neural Networks or more recently the Transformer (you can find two very good papers here, and here). During training, to ensure the same size of the input and output they can just search for the longest sentence they ...

3

The Attention is All you Need has this footnote at the passage motivating the introduction of the $1/\sqrt{d_k}$ factor: To illustrate why the dot products get large, assume that the components of $q$ and $k$ are independent random variables with mean 0 and variance 1. Then their dot product, $q \cdot k = \sum^{d_k}_{i=1}q_ik_i$ has mean 0 and variance $d_k$...

2

As you said, one way to look at it is definitely that the LSTM-encoder's encoding can be only understood by itself, that's why the decoder exists there. An optimisation process encoded it, why couldn't an optimisation process decode it? The hidden state is essentially just an encoding of the information you gave it keeping the time-dependencies in check. ...

2

One renowned example for the specified case is SeqGAN Modeling the data generator as a stochastic policy in reinforcement learning (RL), SeqGAN bypasses the generator differentiation problem by directly performing gradient policy update. The RL reward signal comes from the GAN discriminator judged on a complete sequence, and is passed back to the ...

1

That drawing it's a bit oversimplified. Check this blog for a better explanation and implementation details. I'll refer to the image they have to answer: the yellow boxes represent embedding layers, required to convert words in numbers the green boxes represent the unfolded encoder the red box represent the context vector, i.e. the vector you're looking for....

1

The trained model predicts the probability of a given sequence of tokens. Whatever NLP task you are doing, you usually want to get a high-probability sample from that probability distribution. This sampling task could be quite non-trivial. What you are seeing is most likely the result of a greedy sampling - the most probable next word is chosen from the ...

1

Seq2Seq and Bidirectional RNNs are not doing the same thing, at least in their classic form. Seq2Seq models are used to generate a sequence from another sequence. Consider, for example, the translation task from one language to another. In that sense, Seq2Seq is more a family of models, not an architecture. On the other hand, the Bidirectional RNN is a ...

Only top voted, non community-wiki answers of a minimum length are eligible