# Tag Info

5

I think there are two parts to answering this question. First, about the specific paper that has been mentioned. The paper's title is hyperbolic, and probably written that way to get more people to read it. The paper itself does not make the claim that attention-based networks will supplant existing recurrent network architectures. Instead, it makes a more ...

3

Take the sentence that was generated by your LSTM and feed it back into the LSTM as input. Then the LSTM will generate the next sentence. So the LSTM is using it's previous output as it's input. That's what makes it recursive. The intial word is just your base case. Also you should consider using GPT2 by open AI to do this. It's pretty impressive. https://...

3

You can split each polygon into a collection of triangles and sum up the areas. Not really sure why you would bother with ML. Anyway if you approximates these polygons as images you could maybe train a CNN. Look at the image classification networks which provide bounding boxes.

3

In seq2seq they model the joint distribution of whatever char/word sequence by decomposing it into time-forward conditionals: \begin{align*} p(w_1,w_2,...,w_n) =& \ p(w_1)*p(w_2|w_1) * \ ... \ * p(w_n|w_1,...,w_{n-1}) \\ =& \ p(w_1)*\prod_{i=2}^{n}p(w_i|w_{<i}) \end{align*} This can be sampled by samping each of the conditional is ascending ...

2

As you know, an LSTM language model takes in the past word and tries to predict the new one and continue over a loop. A sentence is divided into tokens and depending on different method, the tokens are divided differently. Some model maybe character based models which simply uses each character as input and output. In this case you can treat punctuation as ...

2

Can the decoder in a transformer model be parallelized like the encoder? NO: Your understanding is completely right. In the decoder, the output of each step is fed to the bottom decoder in the next time step, just like an LSTM. Also, like in LSTMs, the self-attention layer needs to attend to earlier positions in the output sequence in order to compute ...

1

In computational learning theory, the VC dimension is a formal measure of the capacity of a model. The VC dimension is defined in terms of the concept of shattering, so have a look at the related Wikipedia article, which briefly describes the fundamental concept of shattering. See also my answer to the question How to estimate the capacity of a neural ...

1

If you give it a name, it will probably be almost perfect model, as the number of hidden units are definitely enough to store the raw data of a name. However as a neural network it will still not be perfect. There may also be serious overfitting if you do this. A Seq2Seq model have way more parameters than necessary to just "remembers" all the possible names,...

1

He likely found this to a be a best practice to avoid over fitting, with a small data set if you only use small and easy to learn (less words -> less degrees of freedom) sequences then you open your model to the risk of over fitting that data set where as on a large data set that has alot more total information you can train on small sequences without being ...

1

Shouldn't the hidden state of the RNN from the previous timestamps be enough? It is theoretically enough to generate a sequence. However, allowing an input offers a couple of convenient extras: Training data for output sequences is used twice - once as input (as previous sequence data), once as target (to establish loss metric). This may help training ...

1

Can the decoder in a transformer model be parallelized like the encoder? The correct answer is: computation in a Transformer decoder can be parallelized during training, but not during actual translation. What exactly is parallelized? Also, it's worth mentioning that "parallelization" in this case means to compute encoder or decoder states in paralllel ...

Only top voted, non community-wiki answers of a minimum length are eligible