Questions tagged [sequence-modeling]

Questions about the analysis of sequential data, often used to analyse audio information or to predict time series.

Filter by
Sorted by
Tagged with
0 votes
0 answers
65 views

Does the accuracy of a regression learner depend on the way we feed data?

Consider a plot of points as such: As one notices, this looks like an alternating sequence. Further, it can be divided into two subsequences as $a_{\text{odd}}$ and $a_{\text{even}}$ as they seem to ...
DatBoi's user avatar
  • 101
0 votes
0 answers
41 views

VAE for Motion Sequence Generation - Convergence Issue with Scheduled Sampling

I have implemented a Variational Autoencoder (VAE) in PyTorch for motion sequence generation using human pose data (joint angles and angular velocities in radians) from the CMU dataset. The VAE ...
RTn's user avatar
  • 1
1 vote
1 answer
134 views

Can transformers autoregressively generate a sequence of embeddings (instead of predictions)?

Is it theoretically possible to use a transformer architecture to autoregressively generate a sequence of embedding vectors, instead of discrete tokens? For example, if I were to provide an input of a ...
Theo Coombes's user avatar
1 vote
2 answers
148 views

How is the padding mask incorporated in the attention formula?

I have been looking for the answer in other questions but no one tackled that. I want to ask you how is the padding mask considered in the formula of attention? The attention formula taking into ...
Daviiid's user avatar
  • 573
1 vote
1 answer
71 views

Is the problem of Language Modelling a Well-Posed Learning Problem?

Hadamard defines (Well-posed problem (Wikipedia)) a well-posed problem as one for which: a solution exists, the solution is unique, the solution depends continuously on the data (e.g. it is stable) ...
aren't eistert's user avatar
1 vote
0 answers
41 views

The model's accuracy becomes suddenly so unreasonably good at beginning of the training process. I need an explaination

I am practicing machine translation using seq2seq model (more specifically with GRU/LSTM units). The following is my first model: This model first archived about 0.03 accuracy score and gradually ...
Đạt Trần's user avatar
3 votes
1 answer
372 views

Difference between dot product attention and "matrix attention"

As far as I know, attention was first introduced in Learning To Align And Translate. There, the core mechanism which is able to disregard the sequence length, is a dynamically-built matrix, of shape ...
Gulzar's user avatar
  • 749
-1 votes
1 answer
152 views

Understanding self attention - How come there is no connection between different states?

During trying to understand transformers by reading Attention is all you need, I noticed the authors constantly refer to "self attention" without explaining it. The original attention ...
Gulzar's user avatar
  • 749
0 votes
0 answers
140 views

Increasing "output_sequence_length" in TextVectorization layer worsens model's performance

When exploring the Twitter Sentiment Analysis dataset on Kaggle, I came up with a model that looks like this: ...
Tran Khanh's user avatar
1 vote
1 answer
36 views

Many To One LSTM - Can I Use the Same Sequence as Input from Previous Timesteps?

I'm new to LSTMs, and I'm trying to do a basic timeseries prediction using stock prices. However, I'm a bit confused as to how the LSTM is supposed to remember outputs from previous timesteps when it ...
Krusty the Clown's user avatar
0 votes
2 answers
60 views

Sequenatial data: Which model should I apply?

I need to predict a binary vector given a sequential dataset meaning the current datapoint depends on its predecessors as well as (known) successors. So, it looks something like this: Given the ...
toom's user avatar
  • 101
0 votes
1 answer
80 views

Limitations of LSTMs

I'm training an LSTM model for classification on accelerometer data, and I get better results when I downsample the signal to 25 Hz than when I use a 50 Hz signal. I use the same time frame of 1.5 ...
Haffi112's user avatar
  • 121
1 vote
1 answer
55 views

Make an NN utilize other NNs as part of its decision process

Suppose I have a NN that learns to predict the time it takes a robot to move between two jobs. That's three inputs (for starters): robot, job A, job B. Not all robots travel at the same speed, and ...
Brannon's user avatar
  • 111
0 votes
1 answer
537 views

1D Sequence Classification with self-supervised learning

I am working on a multi-class classification task on long one-dimensional sequences. The sequence length may vary in the range $[512, 30720]$, and there is one feature associated each time-step in the ...
Kevin's user avatar
  • 101
1 vote
2 answers
210 views

Is there any reason for giving an index to a token based on its frequency in the text?

In pre-processing of text, we need to assign a number for each token in a text. Then only we can pass it to a model. In pre-processing of text, we need to assign a number for each token in a text. The ...
hanugm's user avatar
  • 3,622
0 votes
1 answer
44 views

What does it mean by "dynamics of a sequence" mathematically?

Consider the following paragraph from the topic named sequential models from the textbook titled Dive into Deep Learning Both cases raise the obvious question of how to generate training data. One ...
hanugm's user avatar
  • 3,622
2 votes
1 answer
1k views

Sequence Embedding using embedding layer: how does the network architecture influence it?

I want to obtain a dense vector representation of protein sequences so that I can meaningfully represent them in an embedding space. We can consider them as sequences of letters, in particular there ...
HelpNeederStudent's user avatar
1 vote
0 answers
162 views

How to compute the loss for a sequence labeling task without the Softmax distribution?

For a sequence labeling task (NER), we compute the loss by passing the softmax distribution of the classes (e.g. vocabulary) with the gold label to the loss function (...
Minions's user avatar
  • 123
5 votes
2 answers
2k views

Why do Transformers have a sequence limit at inference time?

As far as I understand, Transformer's time complexity increases quadratically with respect to the sequence length. As a result, during training to make training feasible, a maximum sequence limit is ...
chessprogrammer's user avatar
8 votes
4 answers
14k views

How can I predict the next number in a non-obvious sequence?

I've got an array of integers ranging from -3 to +3. Example: [1, 3, -2, 0, 0, 1] The array has no obvious pattern since it represents bipolar disorder mood swings. What is the most suitable approach ...
ZenBerry's user avatar
  • 183
1 vote
0 answers
138 views

Couldn't the self-attention mechanism be replaced with a global depth-wise convolution?

The main advantages of the self-attention mechanism are: Ability to capture long-range dependencies Ease to parallelize on GPU or TPU However, I wonder why the same goals cannot be achieved by ...
spiridon_the_sun_rotator's user avatar
3 votes
1 answer
614 views

Is seq2seq the best model when input/output sequences have fixed length?

I understand that seq2seq models are perfectly suitable when the input and/or the output have variable lengths. However, if we know exactly the input/output sequence lengths of the neural network. Is ...
Petrus's user avatar
  • 31
2 votes
1 answer
2k views

What is the proper way to process continuous sequence data, such as time-series, using the Transformer?

What is the right way to input continuous, temporal (time-series) data into the Transformer? Assume we're using the basic TransformerBlock here. Since data is continuous with no tokens, Token ...
meliksahturker's user avatar
0 votes
0 answers
148 views

What is the difference between model setup, model configuration, and model customization?

In the context of research papers related to deep learning models, the authors usually mention these terms in the experiment section when they are talking about the model: configuration, setup. For ...
Ahmad's user avatar
  • 103
4 votes
1 answer
143 views

Can Reinforcement Learning be used to generate sequences?

Can we use reinforcement learning for sequence-to-sequence tasks? If yes, whether or not this is a good choice, how could this be done?
penguin_smasher's user avatar
2 votes
0 answers
36 views

Are there any successful applications of transformers of small size (<10k weights)?

In the problems of NLP and sequence modeling, the Transformer architectures based on the self-attention mechanism (proposed in Attention Is All You Need) have achieved impressive results and now are ...
spiridon_the_sun_rotator's user avatar
0 votes
1 answer
35 views

Normalization of possibly not fully representative data

I am trying to train a classification RNN model on a sequence of table medical data, but I stuck with the normalization problem. I realized that I cannot simply use MinMaxScaler, because of 3 problems:...
banderlog013's user avatar
1 vote
0 answers
57 views

Representing variable-length sequences

I want to train a model over variable-length sequential data (e.g. the temperature at different times of day) where the output depends on what the temperature is at a time ...
Gili's user avatar
  • 111
1 vote
0 answers
41 views

Building a resume recommendation for a job post?

There are few challenges I am facing when building a resume recommendation for a particular job positing. Let's say we convert the resume into a vector on n-dimensions and job description also as an n-...
user_12's user avatar
  • 149
1 vote
1 answer
1k views

Can we use transformers for audio classification tasks?

Since transformers are good at processing sequential data, can we also use them for audio classification problems (same as RNNs)?
Isuru Sachitha's user avatar
2 votes
1 answer
255 views

Mining repeated subsequences in a given sequence

Given an alphabet $I=\left\{i_1,i_2,\dots,i_n\right\}$ and a sequence $S=[e_1,e_2,\dots,e_m]$, where items $e_j \in I$, I am interested in finding every single pattern (subsequence of $S$) that ...
Kikolo's user avatar
  • 73
2 votes
0 answers
125 views

NLP Bible verse division problem: Whats the best model/method?

I'm working on a project compiling various versions of the Bible into a dataset. For the most part versions separate verses discreetly. In some versions, however, verses are combined. Instead of verse ...
rwreed's user avatar
  • 121
1 vote
0 answers
38 views

What's the difference between RNNs and Feed Forward Neural Networks if a fixed size vector can preserve sequential information?

I was watching a Youtube video in which the problem of trying to predict the last word in a sentence was posed. The sentence was "I took my cat for a" and the last word was "walk"....
Prabhav Arora's user avatar
0 votes
0 answers
37 views

Network design to learn multiple sequences of multiple categories

For learning a single sequence, LSTM only should suffice. However, my situation is different here. I have a list of sequences to learn: The sale volumes of 12 months, these are the sequences And ...
Dan D.'s user avatar
  • 1,283
0 votes
1 answer
925 views

Number of LSTM layers needed to learn a certain number of sequences

Theoretically, number of units for a LSTM layer is the number of hidden states or the max length of sequences as per my practice. For example, in Keras: ...
Dan D.'s user avatar
  • 1,283
0 votes
1 answer
77 views

Text classification of non-equal length texts, should I pad left or right?

Text classification of equal length texts works without padding, but in reality, practically, texts never have the same length. For example, spam filtering on blog article: ...
Dan D.'s user avatar
  • 1,283
2 votes
1 answer
204 views

Do transformers have success in other domains different than NLP?

Everybody knows how successful transformers have been in NLP. Is there known work on other domains (e.g that also have a sequential natural way of occurring, such as stock price prediction or other ...
Schach21's user avatar
  • 242
2 votes
1 answer
122 views

How do LSTM or GRU gates learn to specialize in their desired tasks?

While I was studying the equations for the computation inside GRU and LSTM units, I realized that although the different gates have different Weight matrices, their overall structure is the same. They ...
Tolga Aktas's user avatar
4 votes
1 answer
604 views

How can I use machine learning to predict properties (such as the area) of simple polygons?

Imagine a set of simple (non-self-intersecting) polygons given by the coordinate pairs of their vertices $[(x_1, y_1), (x_2, y_2), \dots,(x_n, y_n)]$. The polygons in the set have a different number ...
Vladislav Gladkikh's user avatar
1 vote
0 answers
33 views

length independent sequence classification methods

I am looking to do sequence classification using deep learning. The length of my sequences can vary from a few hundred to several tens of thousands of characters. I was wondering what is a good ...
Sonia's user avatar
  • 11
2 votes
1 answer
403 views

How does the number of stacked LSTM layers or units in each layer affect the model complexity?

I playing around sequence modeling to forecast the weather using LSTM. How does the number of layers or units in each layer exactly affect the model complexity (in an LSTM)? For example, if I ...
Manojk07's user avatar
  • 121
1 vote
0 answers
69 views

Model for supervised sequence classification task

The Problem I am currently working on a sequence classification problem I try to solve with machine learning. The target variable is the current state of a system. This target variable is following a ...
DoKi's user avatar
  • 31
1 vote
0 answers
38 views

What's the best method to predict/generate signal from one sensor (source) to signal from another another (target)?

I was wondering what is the best method out there to find relationship between two 1D signals so that I can predict/generate one (source) from the other (target). For example, let's say that in ...
apples_of_doom's user avatar
1 vote
0 answers
75 views

How to pad sequences during training for an encoder decoder model

I've got an encoder-decoder model for character level English language spelling correction, it is pretty basic stuff with a two LSTM encoder and another LSTM decoder. However, up until now, I have ...
hhaefliger's user avatar
1 vote
0 answers
51 views

Generation of realistic real-valued sequences using Wasserstein GAN fails

My goal is to generate artificial sequences of real-valued data (e.g. time series) with GANs. Starting simple I tried to generate realistic sine-waves using a Wasserstein GAN. But even on this simple ...
needRhelp's user avatar
  • 111
3 votes
2 answers
667 views

How to use LSTM to generate a paragraph

A LSTM model can be trained to generate text sequences by feeding the first word. After feeding the first word, the model will generate a sequence of words (a sentence). Feed the first word to get the ...
Dan D.'s user avatar
  • 1,283
4 votes
0 answers
63 views

Can sequence-to-sequence models be used to convert source code from one programming language to another?

Sequence-to-sequence models have achieved good performance in natural language translation. Could these models also be applied to convert source code written in one programming language to source code ...
pfds2222's user avatar
1 vote
1 answer
39 views

Can a character-level Seq2Seq setup learn to perfectly reconstruct structured data like name strings?

If not perfect, how well can they do? For example, if I give the Seq2Seq setup a name it did not see in the training process, can it output the same name without error? Example ...
ADSBJason's user avatar
1 vote
0 answers
40 views

Sequence-to-Sequence models without specifying the start and end of sentences

Is there a seq-to-seq model which does not require to know the start and end of a sentence? I need to model a system which gets a long sequence of words and creates a long sequence of tokens as long ...
Atena's user avatar
  • 131
6 votes
2 answers
2k views

What evaluation metric are used for sequence-to-sequence prediction problems?

I am solving many sequence-to-sequence prediction problems using RNN/LSTM. What type of evaluation metrics can be used for sequence prediction problems? One metric is the mean squared error (MSE) ...
Asif Khan's user avatar
  • 181