Questions tagged [sequence-modelling]
The sequence-modelling tag has no usage guidance.
23
questions
2
votes
0answers
24 views
How can I prevent a Recursive Neural Network from performing extremely poorly after a few cycles?
I've trained a neural network that can predict the $(n+1)^{th}$ element in a sequence, given the $n^{th}$ element.
It does a pretty good job doing this, with very little error.
The problem emerges ...
1
vote
1answer
16 views
Can a character-level Seq2Seq setup learn to perfectly reconstruct structured data like name strings?
If not perfect, how well can they do? For example, if I give the Seq2Seq setup a name it did not see in the training process, can it output the same name without error?
Example
...
0
votes
0answers
20 views
Sequence-to-Sequence models without specifying the start and end of sentences
Is there a seq-to-seq model which does not require to know the start and end of a sentence? I need to model a system which gets a long sequence of words and creates a long sequence of tokens as long ...
3
votes
0answers
18 views
What evaluation metric are used for sequence-to-sequence prediction problems?
I am solving many sequence-to-sequence prediction problems using RNN/LSTM.
What type of evaluation metrics can be used for sequence prediction problems?
One metric is the mean squared error (MSE) ...
5
votes
1answer
62 views
Why do small datasets require more samples, while big datasets require fewer samples in negative sampling?
In the deep learning specialization course by Andrew Ng, in the video Sequence Models (minute 4:13), he says that in negative sampling we have to choose a sample of words from the corpus to train ...
2
votes
0answers
57 views
Why is the transformer for time series forecasting faster than RNN?
I've been reading different papers which implements the Transformer for time series forecasting. Most of the them are claiming that the training time is significantly faster then using a normal RNN. ...
2
votes
0answers
28 views
Best approach for online Machine Translation with few hundred of samples?
I want to implement a model that improves itself with the passage of time.
My main task is to build a machine translator (from English to Urdu).. The problem I am facing is that I have very little ...
0
votes
0answers
10 views
Why word embedding such as word2vec is not used as the output layer of a seq2seq decoder?
It would make sense to make the decoder predict a smaller embedding vector instead of softmax over a large dictionary.
The word having the most cosine similarity with the output embedding could be ...
0
votes
0answers
15 views
Predict the next best action based on previous lists of actions
I have the following problem. There is a software that I've written some time ago. Users enter customer's data in the system and there is a limited number of things (actions) that they can do/add - ...
0
votes
0answers
14 views
Make an LSTM model for each class separately
I have a dataset of some activities. The dataset contains the status of different sensors and the label of activity. T trained a model in Keras with the following architecture which models the ...
1
vote
1answer
29 views
Is “dataset size” and “model size” same thing?
I mean what is determine my model size, connection amount between layers and neurons, or size of my dataset?
1
vote
0answers
20 views
Literature on Sequence Regresssion
I have some rated time-sequential data and I would like to test if an ANN can learn a correlation between my measurements and ratings.
I suspect I could just try a CNN where 1 Dimension is time or an ...
1
vote
0answers
17 views
Language Model from missing data
I want to learn how a set of operations (my vocabulary) are composed in a dataset of algorithms (corpus).
The algorithms are a sequence of higher level operations which have varying low-level ...
0
votes
1answer
30 views
The Nature of Model Weights for Targeted Dropout
I am trying to figure out how to target certain model weights withtin my 1000x 1000 feed forward network in keras
...
6
votes
3answers
73 views
In sequence-to-sequence, why is the output of the decoder used as its input?
The basic seq-2-seq model consists of 2 parts: a recurrent encoder that compresses a sequence to a vector and decoder that unrolls the vector into the output sequence:
Why is the output, ...
1
vote
1answer
36 views
Can HMM, MRF, or CRF be used to classify the state of a single observation, not the entire observation sequence?
I learn that the Viterbi algorithm used for Hidden Markov Model (HMM) can classify a sequence of hidden states from the corresponding observations; Markov Random Field (MRF) and Conditional Random ...
1
vote
1answer
33 views
Why feeding the correct output as input during training of seq2seq models?
So, I've read about seq2seq for time-series and it seemed really promising, but when I went to implement it, all the tutorial I've found use the correct output as input to the decoder phase during ...
0
votes
0answers
106 views
How Seq2Seq with Bidirectional RNN works?
First of all the scope of the question is as follows - we have Sequence2Sequence architecture with:
Decoder: Bidirectional LSTM
Encoder: regular (single directional) LSTM
What I know:
When you ...
1
vote
1answer
107 views
Why can we approximate the joint probability distribution using the output vector of an LSTM?
In the paper, Contextual String Embeddings for Sequence Labeling, the authors state that
\begin{equation}
P(x_{0:T}) = \prod_{t=0}^T P(x_t|x_{0:t-1})
\end{equation}
They also state that, in the LSTM ...
4
votes
1answer
62 views
Can the decoder in a transformer model be parallelized like the encoder?
Can the decoder in a transformer model be parallelized like the encoder?
As far as I understand, the encoder has all the tokens in the sequence to compute the self-attention scores. But for a decoder,...
2
votes
0answers
107 views
Convolutional Sequence to Sequence Learning: Training vs Generation
I am struggling to understand the use of the Convolutional Sequence to Sequence (Conv-Seq2Seq) model. The image below is take directly from the paper and is the nearly canonical diagram of the ...
1
vote
1answer
218 views
Why do we need both encoder and decoder in sequence to sequence prediction?
Why do we need both encoder and decoder in sequence to sequence prediction?
We could just have a single RNN that, given input $x$, outputs some value $y(t)$ and hidden state $h(t)$. Next, given $h(t)$...
4
votes
1answer
808 views
Will attention based networks prevail over RNN and LSTM?
There is no point in picking one of the growing number of articles that come up in a web search for, "Deep learning attention networks," however the bold claims in Attention Is All You Need, Ashish ...
