Questions tagged [recurrent-neural-networks]

For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.

Filter by
Sorted by
Tagged with
0
votes
0answers
11 views

Is the number of layers in the simple RNN fixed or is it random? [closed]

When I search for RNN, I find LSTM most of the times. Before I go on o for reading more about LSTM, I want to explore the vanilla RNN. I want to know an explanation of working with mathematical ...
2
votes
0answers
23 views

How does backpropagation work in LSTMs?

After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in ...
0
votes
0answers
11 views

Is the number of bidirectional LSTMs in seq2seq model equal to the maximum length of input text/characters?

I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/. It ...
0
votes
0answers
15 views

My LSTM text classification model seems not learn anything in early epochs

I am trying to use LSTM to do text classification and monitor the training process with tensorboard. But it seems that this model doesn't learn anything in early epochs. Is it normal for LSTM networks?...
2
votes
0answers
16 views

Can you use transformer models to do autocomplete tasks?

I've researched online and seen many papers on the use of RNNs (like LSTMs or GRUs) to autocomplete for, say, a search engine, character by character. Which makes sense since it inherently predicts ...
1
vote
0answers
8 views

How do CNNs or RNNs “stack the feature of nodes by a specific order”?

I am trying to understand the following statement taken from the paper Graph Neural Networks: A Review of Methods and Applications (2019). Standard neural networks like CNNs and RNNs cannot handle ...
0
votes
0answers
7 views

How many spectrogram frames per input character does text-to-speech (TTS) system Tacotron-2 generate?

I've been reading on Tacotron-2, a text-to-speech system, that generates speech just-like humans (indistinguishable from humans) using the GitHub https://github.com/Rayhane-mamah/Tacotron-2. I'm very ...
2
votes
0answers
12 views

Reservoir of LSM vs. FF-NN or ELM

The reservoir of the Liquid State Machine is an array of random numbers connected to each other with a probability depending on the distance between each other. Because of this connection with each ...
0
votes
0answers
15 views

NumHiddenUnits in LSTM [migrated]

When we create an LSTM layer in matlab then we specify numHiddenUnits as ...
2
votes
2answers
59 views

How can Siamese Networks be viewed as RNNs?

"Single-object tracking commonly uses Siamese networks, which can be seen as an RNN unrolled over two time-steps." (from the SQAIR paper) I'm wondering how Siamese networks can be viewed as RNNs, as ...
0
votes
0answers
22 views

RNN classifying targets based on size~

I have implemented a recurrent neural network in keras with the following architecture: custom clustering layers from this tutorial: https://www.dlology.com/blog/how-to-do-unsupervised-clustering-...
2
votes
1answer
71 views

What are modern state-of-the-art solutions in prediction of time-series?

I wanted to ask you about the newest achievements in time series analysis (mostly prediction). What state-of-the-art solutions (as in frameworks, papers, related projects) do you know that can be used ...
2
votes
0answers
23 views

Why Pixel RNN (Row LSTM) can capture triangular contexts?

I'm reading the paper Pixel Recurrent Neural Network. I have a question about Row LSTM. Why Row LSTM can capture triangular contexts? In this paper, the kernel of the one-dimensional convolution ...
2
votes
1answer
32 views

Neural networks with internal dynamics in the state-space form

Neural networks with feedback (Hopfield, Hamming, etc.) differ from ordinary neural networks (multilayer perceptrons, etc.), which turns them into a dynamic element with its own internal dynamics (if ...
2
votes
0answers
46 views

What is the time complexity of the forward pass and back-propagation of the sequence-to-sequence model with and without attention?

I keep looking through the literature, but can't seem to find any information regarding the time complexity of the forward pass and back-propagation of the sequence-to-sequence RNN encoder-decoder ...
1
vote
0answers
24 views

How to derive asymptotic run-time of seq2seq RNN? [duplicate]

I want to know the computational complexity of a sequence to sequence RNN in terms of the input and output length. I was wondering how I might go about deriving this in both training and inference.
1
vote
1answer
49 views

How to train a LSTM with multidimensional data

I am trying to train a LSTM, but I have some problems regarding the data representation and feeding it into the model. My data is a numpy array of three dimensions: One sample consist of a 2D matrix ...
6
votes
3answers
554 views

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I am reading the article How Transformers Work where the author writes Another problem with RNNs, and LSTMs, is that it’s hard to parallelize the work for processing sentences, since you have to ...
2
votes
1answer
29 views

Is this a correct visual representation of a recurrent neural network (RNN)?

This is a picture of a recurrent neural network (RNN) found on a udemy course (Deep Learning A-Z). The axis at the bottom is "time". In a time series problem, each yellow row from left to right ...
1
vote
1answer
16 views

Inner working of Bidirectional RNNs

I'm trying to understand how Bidirectional RNNs work. Specifically, I want to know whether a single cell is used with different states, or two different cells are used, each having independent ...
2
votes
0answers
61 views

How to implement a LSTM for multilabel classification problem?

I would like to develop an LSTM because I have a variable input matrix. I am zero-padding to a specific length of 800. However, I am not sure of how to classify a certain situation when each input ...
1
vote
0answers
14 views

Do RNN solves the need for LSTM and/or multiple states in Deep Q-Learning?

Introduction I am trying to setup a Deep Q-Learning agent. I have looked that the papers Playing Atari with Deep Reinforcement Learning as well as Deep Recurrent Q-Learning for Partially Observable ...
1
vote
0answers
20 views

How do we minimize loss for a single neuron with a feedback?

Suppose we had a series of single-dimensional data points $X = \{x_1, x_2, \dots, x_n \}$, where $n$ is the number of data points and there corresponding output values $T = \{t_1, t_2, \dots, t_n \}$. ...
3
votes
0answers
17 views

Is there any way of generating fixed-length sequences with RNNs?

Is there any way of generating fixed-length sequences with RNNs? I want to tell my character level RNN to generate a name of length 3, 4, 5 and so on. I haven't found anything online like this, but my ...
3
votes
0answers
28 views

How to predict an event (or action) based on a window of time-series measurements?

I have an input vector $X$, which contains a series of measurements within a period, e.g. 100 measurements in 1 sec. The goal is to predict an event, let's say, moving forward, backward or static. I ...
0
votes
2answers
43 views

Which NLP applications are based on recurrent neural networks?

Some of the NLP applications taken from this link NLP Applications: Machine Translation Speech Recognition Sentiment Analysis Question Answering Automatic Summarization Chatbots Market Intelligence ...
1
vote
0answers
49 views

Why do momentum techniques not work well for RNNs?

AFAIK, momentum is quite useful when training CNNs, and can speed-up the training substantially without any drop in validation accuracy. I've recently learned that it is not as helpful for RNNs ...
1
vote
1answer
29 views

Transformer for speech recognition?

(1) Are there examples that transformer have better accuracy than RNN end-to-end model like RNN-transducer for speech recognition? (2) Can transformer be used for online speech recognition which ...
1
vote
1answer
31 views

Pros and Cons of Seq2Seq vs Bidirectional RNN

It seems to me that Seq2Seq models and Bidirectional RNNs try to do the same thing. Is that true? Also, when would you recommend one setup over another? Thanks!
0
votes
0answers
20 views

My CTC loss model's loss stagnates and then outputs only blank characters

I am trying to implement BaiDu's DeepSpeech1 in keras using CTC loss, my code is below: ...
1
vote
0answers
13 views

How to pad sequences during training for an encoder decoder model

I've got an encoder-decoder model for character level English language spelling correction, it is pretty basic stuff with a two LSTM encoder and another LSTM decoder. However, up until now, I have ...
1
vote
0answers
15 views

Recurrent neural Network for survival analyses: Dealing with forecast data as feature which can exceed the number of days untill a event occurs

I am building a Recurrent Neural network (LSTM) for predicting the number of days until a Pollen season starts (when the cumulative of the year exceeds X). One of the features I am including in my ...
1
vote
1answer
493 views

What’s the difference between LSTM and RNN?

What's the difference between LSTM and RNN? I know that RNN is a network layer used in neural networks, but what exactly is an LSTM? Is it also a network layer with the same characteristics?
1
vote
0answers
15 views

When stacking LSTM's, should the hidden units increase?

I'm using Weights and Biases to do some hyperparameter sweeping for a supervised sequence-to-sequence problem I'm working on. One thing I noticed is that the sweeps with a gradually increasing number ...
1
vote
0answers
24 views

What is the difference between an generalised estimating equation and a recurrent neural network?

What is the difference between a generalised estimating equation (GEE) model and a recurrent neural network (RNN) model, in terms of what these two models are doing? Apart from the differences in the ...
1
vote
0answers
15 views

Simple sequential model with LSTM which doesn't converge

I'm actually trying to create a sequential neural network in order to translate a "human" sentence in a "machine" sentence understandable by an algorithm. Like It didn't work, I've try to create a NN ...
1
vote
1answer
13 views

how to produce documents like factset blackline?

factset blackline reports essentially can compare two 10-Q SEC filings and show you the difference between the two documents. It highlights added items in green and removed items in red + ...
0
votes
0answers
10 views

What will be the sequence of steps in a human activity recognition model using LSTM?

In the context of these steps detection, tracking, action classification and activity recognition. Which step will be first and further sequence?
0
votes
0answers
16 views

RNN/LSTM with a large amounts of data

I have sequence data that's quite large - 4x65k per sample. I'm interested in doing classification problems. The number of classes is moderate - ~27 or so What is the suggestion for dealing with this ...
1
vote
1answer
62 views

What is teacher forcing?

In the paper Neural Programmer-Interpreters, the authors use the teacher forcing technique, but what exactly is it?
1
vote
1answer
47 views

Are there names for neural networks with a well-defined layer or neuron characteristics?

Are there names for neural networks with a well-defined layer or neuron characteristics? For example, a matrix that has the same number of rows and columns is called a square matrix. Is there an ...
1
vote
0answers
35 views

Can GraphRNN be used with very large graphs?

In the GraphRNN paper, the authors only implement the algorithm up to a graph size of 2k nodes. Would this still work on much larger graphs (on the order of $10^7$)? Or would the computation just ...
0
votes
0answers
23 views

Web stream requests prediction architecture

What's in your opinion the best possible architecture for the following problem ? If you have any code that can be used it would be great . Dataset : 400.000 samples given in hex format in an .xlsx ...
2
votes
0answers
45 views

How do I train a multiple-speaker model (speech synthesis) based on Tacotron 2 and espnet?

I'm new to Speech Synthesis & Deep Learning. Recently, I got a task as described below: I have problem in training a multi-speaker model which should be created by Tacotron2. And I was told I can ...
1
vote
1answer
20 views

What is the appropriate RNN structure to do Sentiment Analysis with multiple dependent ratings?

Suppose we are doing sentiment analysis for a restaurant. Customers can rate the restaurant by #1: how expensive the restaurant is, #2:how good is the food and #3: how likely they will come again. The ...
1
vote
1answer
21 views

Convert input dataset given in hex addresses to int

I have created an LSTM Neural Network which take as input the following format in an .csv file ...
1
vote
0answers
15 views

Is there a way to use RNN (in tensorflow) to do something like a batch Kalman with the weight dynamics specified in the loss?

Or would you simply do this as a time series of models. Basically I think you can think of time series of weights as the hidden states and the dynamics driving the weight time series as the RNN ...
1
vote
0answers
32 views

How to exploit translational symmetry for extrapolation in video generation using machine learning

I'll try to rephrase my problem in the context of video processing. Imagine that initial frame of video has some translational symmetry. The frame evolves according to an update rule. I generate a ...
0
votes
1answer
87 views

How to process data in a data stream for a LSTM

How can a data stream for a RNN (LSTM) be handled, when the stream contains data sets belonging to different prediction classes? Training phase: I have trained a LSTM to predict a class out of a ...
1
vote
0answers
10 views

How do I build a multi RNN network with keras?

I have 2 (independently long) sequences (a and b) of feature vectors that I want to use as input for a neural network. The idea was to build 2 GRU based encoders (one for each sequence). I would than ...