Questions tagged [sequence-modelling]
The sequence-modelling tag has no usage guidance.
57
questions
1
vote
1
answer
26
views
Make an NN utilize other NNs as part of its decision process
Suppose I have a NN that learns to predict the time it takes a robot to move between two jobs. That's three inputs (for starters): robot, job A, job B. Not all robots travel at the same speed, and ...
- 111
0
votes
0
answers
12
views
Vector to sequence RNNs: do they take a random initial "prompt"?
I am going through the Deep Learning book by Ian Goodfellow (here) and came by the architecture for a vector to sequence RNN (Figure 10.9). I am not sure I understand how this architecture works and ...
- 143
0
votes
1
answer
54
views
1D Sequence Classification with self-supervised learning
I am working on a multi-class classification task on long one-dimensional sequences. The sequence length may vary in the range $[512, 30720]$, and there is one feature associated each time-step in the ...
- 101
0
votes
0
answers
13
views
Deep learning to fill sequence elements missing at random
I have the following problem setup:
There is a list of floats (between -1 and 1) that is about 768*2 in length. The values of the floats are features that depend on two documents, the first 768 ...
1
vote
2
answers
34
views
Is there any reason for giving an index to a token based on its frequency in the text?
In pre-processing of text, we need to assign a number for each token in a text. Then only we can pass it to a model. In pre-processing of text, we need to assign a number for each token in a text. The ...
- 3,131
0
votes
1
answer
35
views
What does it mean by "dynamics of a sequence" mathematically?
Consider the following paragraph from the topic named sequential models from the textbook titled Dive into Deep Learning
Both cases raise the obvious question of how to generate training
data. One ...
- 3,131
1
vote
1
answer
193
views
Sequence Embedding using embedding layer: how does the network architecture influence it?
I want to obtain a dense vector representation of protein sequences so that I can meaningfully represent them in an embedding space. We can consider them as sequences of letters, in particular there ...
1
vote
0
answers
29
views
How to compute the loss for a sequence labeling task without the Softmax distribution?
For a sequence labeling task (NER), we compute the loss by passing the softmax distribution of the classes (e.g. vocabulary) with the gold label to the loss function (...
- 123
2
votes
2
answers
280
views
Why do Transformers have a sequence limit at inference time?
As far as I understand, Transformer's time complexity increases quadratically with respect to the sequence length. As a result, during training to make training feasible, a maximum sequence limit is ...
- 941
0
votes
0
answers
67
views
What would be the total number of learnable parameters of the RNN encoder of this encoder-decoder architecture for machine translation?
Here's a quiz. My answer is different from the teacher's, so I'm wondering what answer would you pick up.
We use a sequence-to-sequence (encoder-decoder) system to perform
machine translation. We ...
- 113
5
votes
4
answers
4k
views
How can I predict the next number in a non-obvious sequence?
I've got an array of integers ranging from -3 to +3.
Example: [1, 3, -2, 0, 0, 1]
The array has no obvious pattern since it represents bipolar disorder mood swings.
What is the most suitable approach ...
- 153
1
vote
0
answers
91
views
Couldn't the self-attention mechanism be replaced with a global depth-wise convolution?
The main advantages of the self-attention mechanism are:
Ability to capture long-range dependencies
Ease to parallelize on GPU or TPU
However, I wonder why the same goals cannot be achieved by ...
- 1,824
1
vote
2
answers
152
views
Is seq2seq the best model when input/output sequences have fixed length?
I understand that seq2seq models are perfectly suitable when the input and/or the output have variable lengths. However, if we know exactly the input/output sequence lengths of the neural network. Is ...
- 11
0
votes
0
answers
15
views
Are there any well-known ways to fuzzy-cluster (variable length) sequences of trajectories?
I have this issue where I need to create 'soft' clusters for different trajectories. The data is sequences of integers where each integer means a specific point; so I have sequences like $s=(1,47,9)$ ...
- 509
1
vote
1
answer
337
views
What is the proper way to process continuous sequence data, such as time-series, using the Transformer?
What is the right way to input continuous, temporal (time-series) data into the Transformer? Assume we're using the basic TransformerBlock here.
Since data is continuous with no tokens, Token ...
- 286
0
votes
0
answers
41
views
What is the difference between model setup, model configuration, and model customization?
In the context of research papers related to deep learning models, the authors usually mention these terms in the experiment section when they are talking about the model: configuration, setup. For ...
- 3
3
votes
1
answer
66
views
Can Reinforcement Learning be used to generate sequences?
Can we use reinforcement learning for sequence-to-sequence tasks? If yes, whether or not this is a good choice, how could this be done?
2
votes
0
answers
27
views
Are there any successful applications of transformers of small size (<10k weights)?
In the problems of NLP and sequence modeling, the Transformer architectures based on the self-attention mechanism (proposed in Attention Is All You Need) have achieved impressive results and now are ...
- 1,824
0
votes
1
answer
31
views
Normalization of possibly not fully representative data
I am trying to train a classification RNN model on a sequence of table medical data, but I stuck with the normalization problem. I realized that I cannot simply use MinMaxScaler, because of 3 problems:...
- 111
1
vote
0
answers
37
views
Representing variable-length sequences
I want to train a model over variable-length sequential data (e.g. the temperature at different times of day) where the output depends on what the temperature is at a time ...
- 111
1
vote
0
answers
26
views
Building a resume recommendation for a job post?
There are few challenges I am facing when building a resume recommendation for a particular job positing.
Let's say we convert the resume into a vector on n-dimensions and job description also as an n-...
- 149
1
vote
1
answer
695
views
Can we use transformers for audio classification tasks?
Since transformers are good at processing sequential data, can we also use them for audio classification problems (same as RNNs)?
2
votes
1
answer
117
views
Mining repeated subsequences in a given sequence
Given an alphabet $I=\left\{i_1,i_2,\dots,i_n\right\}$ and a sequence $S=[e_1,e_2,\dots,e_m]$, where items $e_j \in I$, I am interested in finding every single pattern (subsequence of $S$) that ...
- 73
2
votes
0
answers
80
views
NLP Bible verse division problem: Whats the best model/method?
I'm working on a project compiling various versions of the Bible into a dataset. For the most part versions separate verses discreetly. In some versions, however, verses are combined. Instead of verse ...
- 21
1
vote
0
answers
28
views
What's the difference between RNNs and Feed Forward Neural Networks if a fixed size vector can preserve sequential information?
I was watching a Youtube video in which the problem of trying to predict the last word in a sentence was posed. The sentence was "I took my cat for a" and the last word was "walk"....
0
votes
0
answers
35
views
Network design to learn multiple sequences of multiple categories
For learning a single sequence, LSTM only should suffice.
However, my situation is different here. I have a list of sequences to learn:
The sale volumes of 12 months, these are the sequences
And ...
- 1,183
0
votes
1
answer
368
views
Number of LSTM layers needed to learn a certain number of sequences
Theoretically, number of units for a LSTM layer is the number of hidden states or the max length of sequences as per my practice.
For example, in Keras:
...
- 1,183
0
votes
1
answer
51
views
Text classification of non-equal length texts, should I pad left or right?
Text classification of equal length texts works without padding, but in reality, practically, texts never have the same length.
For example, spam filtering on blog article:
...
- 1,183
2
votes
1
answer
112
views
Do transformers have success in other domains different than NLP?
Everybody knows how successful transformers have been in NLP. Is there known work on other domains (e.g that also have a sequential natural way of occurring, such as stock price prediction or other ...
- 242
2
votes
1
answer
64
views
How do LSTM or GRU gates learn to specialize in their desired tasks?
While I was studying the equations for the computation inside GRU and LSTM units, I realized that although the different gates have different Weight matrices, their overall structure is the same. They ...
- 23
4
votes
1
answer
357
views
How can I use machine learning to predict properties (such as the area) of simple polygons?
Imagine a set of simple (non-self-intersecting) polygons given by the coordinate pairs of their vertices $[(x_1, y_1), (x_2, y_2), \dots,(x_n, y_n)]$. The polygons in the set have a different number ...
1
vote
0
answers
32
views
length independent sequence classification methods
I am looking to do sequence classification using deep learning. The length of my sequences can vary from a few hundred to several tens of thousands of characters. I was wondering what is a good ...
- 11
2
votes
1
answer
221
views
How does the number of stacked LSTM layers or units in each layer affect the model complexity?
I playing around sequence modeling to forecast the weather using LSTM.
How does the number of layers or units in each layer exactly affect the model complexity (in an LSTM)? For example, if I ...
- 121
1
vote
0
answers
58
views
Model for supervised sequence classification task
The Problem
I am currently working on a sequence classification problem I try to solve with machine learning.
The target variable is the current state of a system.
This target variable is following a ...
- 31
1
vote
0
answers
28
views
What's the best method to predict/generate signal from one sensor (source) to signal from another another (target)?
I was wondering what is the best method out there to find relationship between two 1D signals so that I can predict/generate one (source) from the other (target). For example, let's say that in ...
1
vote
0
answers
57
views
How to pad sequences during training for an encoder decoder model
I've got an encoder-decoder model for character level English language spelling correction, it is pretty basic stuff with a two LSTM encoder and another LSTM decoder.
However, up until now, I have ...
- 111
1
vote
0
answers
37
views
Generation of realistic real-valued sequences using Wasserstein GAN fails
My goal is to generate artificial sequences of real-valued data (e.g. time series) with GANs. Starting simple I tried to generate realistic sine-waves using a Wasserstein GAN. But even on this simple ...
- 111
3
votes
2
answers
405
views
How to use LSTM to generate a paragraph
A LSTM model can be trained to generate text sequences by feeding the first word. After feeding the first word, the model will generate a sequence of words (a sentence). Feed the first word to get the ...
- 1,183
3
votes
0
answers
45
views
Can sequence-to-sequence models be used to convert source code from one programming language to another?
Sequence-to-sequence models have achieved good performance in natural language translation. Could these models also be applied to convert source code written in one programming language to source code ...
- 31
1
vote
1
answer
32
views
Can a character-level Seq2Seq setup learn to perfectly reconstruct structured data like name strings?
If not perfect, how well can they do? For example, if I give the Seq2Seq setup a name it did not see in the training process, can it output the same name without error?
Example
...
- 51
1
vote
0
answers
38
views
Sequence-to-Sequence models without specifying the start and end of sentences
Is there a seq-to-seq model which does not require to know the start and end of a sentence? I need to model a system which gets a long sequence of words and creates a long sequence of tokens as long ...
- 131
5
votes
2
answers
1k
views
What evaluation metric are used for sequence-to-sequence prediction problems?
I am solving many sequence-to-sequence prediction problems using RNN/LSTM.
What type of evaluation metrics can be used for sequence prediction problems?
One metric is the mean squared error (MSE) ...
- 171
5
votes
1
answer
124
views
Why do small datasets require more samples, while big datasets require fewer samples in negative sampling?
In the deep learning specialization course by Andrew Ng, in the video Sequence Models (minute 4:13), he says that in negative sampling we have to choose a sample of words from the corpus to train ...
- 61
2
votes
0
answers
252
views
Why is the transformer for time series forecasting faster than RNN?
I've been reading different papers which implements the Transformer for time series forecasting. Most of the them are claiming that the training time is significantly faster then using a normal RNN. ...
- 1,098
2
votes
0
answers
34
views
Best approach for online Machine Translation with few hundred of samples?
I want to implement a model that improves itself with the passage of time.
My main task is to build a machine translator (from English to Urdu).. The problem I am facing is that I have very little ...
- 121
1
vote
1
answer
2k
views
Is "dataset size" and "model size" same thing? [closed]
I mean what is determine my model size, connection amount between layers and neurons, or size of my dataset?
- 21
1
vote
0
answers
24
views
Literature on Sequence Regresssion
I have some rated time-sequential data and I would like to test if an ANN can learn a correlation between my measurements and ratings.
I suspect I could just try a CNN where 1 Dimension is time or an ...
- 153
1
vote
0
answers
23
views
Language Model from missing data
I want to learn how a set of operations (my vocabulary) are composed in a dataset of algorithms (corpus).
The algorithms are a sequence of higher level operations which have varying low-level ...
- 131
6
votes
3
answers
191
views
In sequence-to-sequence, why is the output of the decoder used as its input?
The basic seq-2-seq model consists of 2 parts: a recurrent encoder that compresses a sequence to a vector and decoder that unrolls the vector into the output sequence:
Why is the output, ...
- 348
1
vote
1
answer
119
views
Can HMM, MRF, or CRF be used to classify the state of a single observation, not the entire observation sequence?
I learn that the Viterbi algorithm used for Hidden Markov Model (HMM) can classify a sequence of hidden states from the corresponding observations; Markov Random Field (MRF) and Conditional Random ...
- 288