# Questions tagged [sequence-modelling]

The tag has no usage guidance.

37 questions
Filter by
Sorted by
Tagged with
33 views

### Network design to learn multiple sequences of multiple categories

For learning a single sequence, LSTM only should suffice. However, my situation is different here. I have a list of sequences to learn: The sale volumes of 12 months, these are the sequences And ...
31 views

### Number of LSTM layers needed to learn a certain number of sequences

Theoretically, number of units for a LSTM layer is the number of hidden states or the max length of sequences as per my practice. For example, in Keras: ...
24 views

### Text classification of non-equal length texts, should I pad left or right?

Text classification of equal length texts works without padding, but in reality, practically, texts never have the same length. For example, spam filtering on blog article: ...
35 views

### Do transformers have success in other domains different than NLP?

Everybody knows how successful transformers have been in NLP. Is there known work on other domains (e.g that also have a sequential natural way of occurring, such as stock price prediction or other ...
33 views

### How do LSTM or GRU gates learn to specialize in their desired tasks?

While I was studying the equations for the computation inside GRU and LSTM units, I realized that although the different gates have different Weight matrices, their overall structure is the same. They ...
24 views

### How to derive asymptotic run-time of seq2seq RNN? [duplicate]

I want to know the computational complexity of a sequence to sequence RNN in terms of the input and output length. I was wondering how I might go about deriving this in both training and inference.
67 views

### How can I use machine learning to predict properties (such as the area) of simple polygons?

Imagine a set of simple (non-self-intersecting) polygons given by the coordinate pairs of their vertices $[(x_1, y_1), (x_2, y_2), \dots,(x_n, y_n)]$. The polygons in the set have a different number ...
24 views

### length independent sequence classification methods

I am looking to do sequence classification using deep learning. The length of my sequences can vary from a few hundred to several tens of thousands of characters. I was wondering what is a good ...
57 views

### How does the number of stacked LSTM layers or units in each layer affect the model complexity?

I playing around sequence modeling to forecast the weather using LSTM. How does the number of layers or units in each layer exactly affect the model complexity (in an LSTM)? For example, if I ...
33 views

### Model for supervised sequence classification task

The Problem I am currently working on a sequence classification problem I try to solve with machine learning. The target variable is the current state of a system. This target variable is following a ...
22 views

### What's the best method to predict/generate signal from one sensor (source) to signal from another another (target)?

I was wondering what is the best method out there to find relationship between two 1D signals so that I can predict/generate one (source) from the other (target). For example, let's say that in ...
21 views

### How to pad sequences during training for an encoder decoder model

I've got an encoder-decoder model for character level English language spelling correction, it is pretty basic stuff with a two LSTM encoder and another LSTM decoder. However, up until now, I have ...
15 views

### Generation of realistic real-valued sequences using Wasserstein GAN fails

My goal is to generate artificial sequences of real-valued data (e.g. time series) with GANs. Starting simple I tried to generate realistic sine-waves using a Wasserstein GAN. But even on this simple ...
117 views

### How to use LSTM to generate a paragraph

A LSTM model can be trained to generate text sequences by feeding the first word. After feeding the first word, the model will generate a sequence of words (a sentence). Feed the first word to get the ...
37 views

### Can sequence-to-sequence models be used to convert source code from one programming language to another?

Sequence-to-sequence models have achieved good performance in natural language translation. Could these models also be applied to convert source code written in one programming language to source code ...
36 views

### How can I prevent a Recursive Neural Network from performing extremely poorly after a few cycles?

I've trained a neural network that can predict the $(n+1)^{th}$ element in a sequence, given the $n^{th}$ element. It does a pretty good job doing this, with very little error. The problem emerges ...
25 views

### Can a character-level Seq2Seq setup learn to perfectly reconstruct structured data like name strings?

If not perfect, how well can they do? For example, if I give the Seq2Seq setup a name it did not see in the training process, can it output the same name without error? Example ...
32 views

### Sequence-to-Sequence models without specifying the start and end of sentences

Is there a seq-to-seq model which does not require to know the start and end of a sentence? I need to model a system which gets a long sequence of words and creates a long sequence of tokens as long ...
143 views

### What evaluation metric are used for sequence-to-sequence prediction problems?

I am solving many sequence-to-sequence prediction problems using RNN/LSTM. What type of evaluation metrics can be used for sequence prediction problems? One metric is the mean squared error (MSE) ...
73 views

### Why do small datasets require more samples, while big datasets require fewer samples in negative sampling?

In the deep learning specialization course by Andrew Ng, in the video Sequence Models (minute 4:13), he says that in negative sampling we have to choose a sample of words from the corpus to train ...
141 views

### Why is the transformer for time series forecasting faster than RNN?

I've been reading different papers which implements the Transformer for time series forecasting. Most of the them are claiming that the training time is significantly faster then using a normal RNN. ...
33 views

### Best approach for online Machine Translation with few hundred of samples?

I want to implement a model that improves itself with the passage of time. My main task is to build a machine translator (from English to Urdu).. The problem I am facing is that I have very little ...
14 views

### Why word embedding such as word2vec is not used as the output layer of a seq2seq decoder?

It would make sense to make the decoder predict a smaller embedding vector instead of softmax over a large dictionary. The word having the most cosine similarity with the output embedding could be ...
63 views

### Predict the next best action based on previous lists of actions

I have the following problem. There is a software that I've written some time ago. Users enter customer's data in the system and there is a limited number of things (actions) that they can do/add - ...
16 views

### Make an LSTM model for each class separately

I have a dataset of some activities. The dataset contains the status of different sensors and the label of activity. T trained a model in Keras with the following architecture which models the ...
184 views

### Is “dataset size” and “model size” same thing?

I mean what is determine my model size, connection amount between layers and neurons, or size of my dataset?
22 views

### Literature on Sequence Regresssion

I have some rated time-sequential data and I would like to test if an ANN can learn a correlation between my measurements and ratings. I suspect I could just try a CNN where 1 Dimension is time or an ...
20 views

### Language Model from missing data

I want to learn how a set of operations (my vocabulary) are composed in a dataset of algorithms (corpus). The algorithms are a sequence of higher level operations which have varying low-level ...
34 views

### The Nature of Model Weights for Targeted Dropout

I am trying to figure out how to target certain model weights withtin my 1000x 1000 feed forward network in keras ...
88 views

### In sequence-to-sequence, why is the output of the decoder used as its input?

The basic seq-2-seq model consists of 2 parts: a recurrent encoder that compresses a sequence to a vector and decoder that unrolls the vector into the output sequence: Why is the output, ...
63 views

### Can HMM, MRF, or CRF be used to classify the state of a single observation, not the entire observation sequence?

I learn that the Viterbi algorithm used for Hidden Markov Model (HMM) can classify a sequence of hidden states from the corresponding observations; Markov Random Field (MRF) and Conditional Random ...
44 views

### Why feeding the correct output as input during training of seq2seq models?

So, I've read about seq2seq for time-series and it seemed really promising, but when I went to implement it, all the tutorial I've found use the correct output as input to the decoder phase during ...
169 views

### Why can we approximate the joint probability distribution using the output vector of an LSTM?

In the paper, Contextual String Embeddings for Sequence Labeling, the authors state that \begin{equation} P(x_{0:T}) = \prod_{t=0}^T P(x_t|x_{0:t-1}) \end{equation} They also state that, in the LSTM ...
300 views

### Can the decoder in a transformer model be parallelized like the encoder?

Can the decoder in a transformer model be parallelized like the encoder? As far as I understand, the encoder has all the tokens in the sequence to compute the self-attention scores. But for a decoder,...
185 views

### Convolutional Sequence to Sequence Learning: Training vs Generation

I am struggling to understand the use of the Convolutional Sequence to Sequence (Conv-Seq2Seq) model. The image below is take directly from the paper and is the nearly canonical diagram of the ...
Why do we need both encoder and decoder in sequence to sequence prediction? We could just have a single RNN that, given input $x$, outputs some value $y(t)$ and hidden state $h(t)$. Next, given $h(t)$...