Questions tagged [recurrent-neural-networks]

For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.

Filter by
Sorted by
Tagged with
1
vote
0answers
8 views

How Training of the “Attention model” in “ Attention is all you need” paper done? What are Keys, Values?

I have recently encountered the paper on NLP. It is very new to me and I am still unable to see how that works. I have used all the resources over there from the original paper to Youtube videos and ...
1
vote
0answers
10 views

How to understand the matrices used in the Attention layer?

Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
12
votes
2answers
2k views

What is a recurrent neural network?

Surprisingly, this wasn't asked before - at least I didn't find anything besides some vaguely related questions. So, what is a recurrent neural network, and what are their advantages over regular (or ...
2
votes
0answers
29 views

Why do we need recurrent neural networks instead of feed-forward neural networks? [duplicate]

Why do we need recurrent neural networks instead of feed-forward neural networks? What are the advantages of RNNs compared with FFNNs?
0
votes
0answers
5 views

How to exclude sections of bad data from time-series data before training an LSTM network

I am using LSTM network for predicting IOT time-series data receiving from un-reliable devices and networks. This results in several multiple sections [continuous streak of bad data for several days ...
1
vote
1answer
16 views

How to make a LSTM network to predict sequence only after input sequence is finished?

I am learning to use a LSTM model to predict time series data. Specifically, I hope the network should output a sequence (with multiple time steps) only after the input sequence has finished feeding ...
1
vote
0answers
19 views

Deep learning techniques with time-fixed, time-dependent and imaging data

I have a question about the use of deep learning techniques with time-fixed features and images (setting 1) and time-dependent features (setting 2). (I am pretty new to the deep learning world so ...
1
vote
0answers
25 views

Why do we use a delay when feeding our input data to the echo state network?

I'm new to working with neural networks and have recently began implementing neural networks for time series forecasting in some of my work. I've been particularly using Echo State Networks and have ...
3
votes
2answers
90 views

Are there neural networks that accept graphs or trees as inputs?

As far I know, the RNN accepts a sequence as input and can produce as a sequence as output. Are there neural networks that accept graphs or trees as inputs, so that to represent the relationships ...
0
votes
0answers
12 views

Is the number of layers in the simple RNN fixed or is it random? [closed]

When I search for RNN, I find LSTM most of the times. Before I go on o for reading more about LSTM, I want to explore the vanilla RNN. I want to know an explanation of working with mathematical ...
0
votes
2answers
43 views

Which NLP applications are based on recurrent neural networks?

Some of the NLP applications taken from this link NLP Applications: Machine Translation Speech Recognition Sentiment Analysis Question Answering Automatic Summarization Chatbots Market Intelligence ...
3
votes
0answers
29 views

How does backpropagation work in LSTMs?

After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in ...
0
votes
0answers
11 views

Is the number of bidirectional LSTMs in seq2seq model equal to the maximum length of input text/characters?

I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/. It ...
1
vote
0answers
16 views

My LSTM text classification model seems not learn anything in early epochs

I am trying to use LSTM to do text classification and monitor the training process with tensorboard. But it seems that this model doesn't learn anything in early epochs. Is it normal for LSTM networks?...
2
votes
1answer
73 views

What are modern state-of-the-art solutions in prediction of time-series?

I wanted to ask you about the newest achievements in time series analysis (mostly prediction). What state-of-the-art solutions (as in frameworks, papers, related projects) do you know that can be used ...
2
votes
0answers
17 views

Can you use transformer models to do autocomplete tasks?

I've researched online and seen many papers on the use of RNNs (like LSTMs or GRUs) to autocomplete for, say, a search engine, character by character. Which makes sense since it inherently predicts ...
1
vote
0answers
8 views

How do CNNs or RNNs “stack the feature of nodes by a specific order”?

I am trying to understand the following statement taken from the paper Graph Neural Networks: A Review of Methods and Applications (2019). Standard neural networks like CNNs and RNNs cannot handle ...
0
votes
0answers
7 views

How many spectrogram frames per input character does text-to-speech (TTS) system Tacotron-2 generate?

I've been reading on Tacotron-2, a text-to-speech system, that generates speech just-like humans (indistinguishable from humans) using the GitHub https://github.com/Rayhane-mamah/Tacotron-2. I'm very ...
4
votes
3answers
109 views

Can we optimize an optimization algorithm?

In this answer to the question Is an optimization algorithm equivalent to a neural network?, the author stated that, in theory, there is some recurrent neural network that implements a given ...
5
votes
1answer
539 views

What is the relationship between the size of the hidden layer and the size of the cell state layer in an LSTM?

I was following some examples to get familiar with TensorFlow's LSTM API, but noticed that all LSTM initialization functions require only the num_units parameter, ...
2
votes
0answers
13 views

Reservoir of LSM vs. FF-NN or ELM

The reservoir of the Liquid State Machine is an array of random numbers connected to each other with a probability depending on the distance between each other. Because of this connection with each ...
1
vote
2answers
142 views

Which algorithm should I use to map an input sentence to an output sentence?

I am new to NLP realm. If you have an input text "The price of orange has increased" and output text "Increase the production of orange". Can we make our RNN model to predict the output text? Or what ...
2
votes
2answers
59 views

How can Siamese Networks be viewed as RNNs?

"Single-object tracking commonly uses Siamese networks, which can be seen as an RNN unrolled over two time-steps." (from the SQAIR paper) I'm wondering how Siamese networks can be viewed as RNNs, as ...
3
votes
1answer
56 views

How are the observations stored in the RNN that encodes the state?

I am a bit confused about observations in RL systems which use RNN to encode the state. I read a few papers like this and this. If I were to use a sequence of raw observations (or features) as an ...
0
votes
0answers
15 views
3
votes
2answers
310 views

What is the significance of this Stanford University “Financial Market Time Series Prediction with RNN's” paper?

Researchers at Stanford University released this paper in 2012: http://cs229.stanford.edu/proj2012/BernalFokPidaparthi-FinancialMarketTimeSeriesPredictionwithRecurrentNeural.pdf It goes on to ...
0
votes
0answers
22 views

RNN classifying targets based on size~

I have implemented a recurrent neural network in keras with the following architecture: custom clustering layers from this tutorial: https://www.dlology.com/blog/how-to-do-unsupervised-clustering-...
6
votes
3answers
764 views

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I am reading the article How Transformers Work where the author writes Another problem with RNNs, and LSTMs, is that it’s hard to parallelize the work for processing sentences, since you have to ...
1
vote
1answer
66 views

What is teacher forcing?

In the paper Neural Programmer-Interpreters, the authors use the teacher forcing technique, but what exactly is it?
2
votes
0answers
23 views

Why Pixel RNN (Row LSTM) can capture triangular contexts?

I'm reading the paper Pixel Recurrent Neural Network. I have a question about Row LSTM. Why Row LSTM can capture triangular contexts? In this paper, the kernel of the one-dimensional convolution ...
2
votes
1answer
32 views

Neural networks with internal dynamics in the state-space form

Neural networks with feedback (Hopfield, Hamming, etc.) differ from ordinary neural networks (multilayer perceptrons, etc.), which turns them into a dynamic element with its own internal dynamics (if ...
2
votes
1answer
265 views

How can active learning be used in the case of complex models that require a lot of data?

We have a series of data and we want to label the parts of each series. As we do not have any training data, we could try to use active learning as a solution, but the problem is that our classifier ...
14
votes
2answers
260 views

How do I decide the optimal number of layers for a neural network?

How do I decide the optimal number of layers for a neural network (feedforward or recurrent)?
4
votes
1answer
1k views

How can I understand this statement about RNNs and hidden layers?

In the lecture, there was a statement: Recurrent neural networks with multiple hidden layers are just a special case that has some of the hidden to hidden connections missing. I understand ...
2
votes
1answer
292 views

How do I choose the size of the hidden state of a GRU?

I'm trying to understand how the size of the hidden state affects the GRU. For example, suppose I want to make a GRU count. I'm gonna feed it with three numbers, and I expect it to predict the ...
10
votes
4answers
787 views

What are the models that have the potential to replace neural networks in the near future?

Are there possible models that have the potential to replace neural networks in the near future? And do we even need that? What is the worst thing about using neural networks in terms of efficiency?
3
votes
0answers
143 views

Training RNN's on text: Can you use an ASCII encoding just as well as a one-hot character encoding?

I've mostly seen (e.g. in The Unreasonable Effectiveness of Recurrent Neural Networks) that when training RNN on text for something like language modeling, the text is usually featurized character-by-...
1
vote
1answer
55 views

How to train a LSTM with multidimensional data

I am trying to train a LSTM, but I have some problems regarding the data representation and feeding it into the model. My data is a numpy array of three dimensions: One sample consist of a 2D matrix ...
2
votes
0answers
47 views

What is the time complexity of the forward pass and back-propagation of the sequence-to-sequence model with and without attention?

I keep looking through the literature, but can't seem to find any information regarding the time complexity of the forward pass and back-propagation of the sequence-to-sequence RNN encoder-decoder ...
1
vote
0answers
24 views

How to derive asymptotic run-time of seq2seq RNN? [duplicate]

I want to know the computational complexity of a sequence to sequence RNN in terms of the input and output length. I was wondering how I might go about deriving this in both training and inference.
2
votes
0answers
61 views

How to implement a LSTM for multilabel classification problem?

I would like to develop an LSTM because I have a variable input matrix. I am zero-padding to a specific length of 800. However, I am not sure of how to classify a certain situation when each input ...
1
vote
1answer
16 views

Inner working of Bidirectional RNNs

I'm trying to understand how Bidirectional RNNs work. Specifically, I want to know whether a single cell is used with different states, or two different cells are used, each having independent ...
2
votes
1answer
29 views

Is this a correct visual representation of a recurrent neural network (RNN)?

This is a picture of a recurrent neural network (RNN) found on a udemy course (Deep Learning A-Z). The axis at the bottom is "time". In a time series problem, each yellow row from left to right ...
1
vote
1answer
34 views

Pros and Cons of Seq2Seq vs Bidirectional RNN

It seems to me that Seq2Seq models and Bidirectional RNNs try to do the same thing. Is that true? Also, when would you recommend one setup over another? Thanks!
4
votes
1answer
452 views

Over- and underestimations of the lowest and highest values in LSTM network

I'm training a LSTM network with multiple inputs and several LSTM layers in order to setup a time series gap filling procedure. The LSTM is trained bidirectionally with "tanh" activation on the ...
1
vote
0answers
20 views

How do we minimize loss for a single neuron with a feedback?

Suppose we had a series of single-dimensional data points $X = \{x_1, x_2, \dots, x_n \}$, where $n$ is the number of data points and there corresponding output values $T = \{t_1, t_2, \dots, t_n \}$. ...
1
vote
0answers
15 views

Do RNN solves the need for LSTM and/or multiple states in Deep Q-Learning?

Introduction I am trying to setup a Deep Q-Learning agent. I have looked that the papers Playing Atari with Deep Reinforcement Learning as well as Deep Recurrent Q-Learning for Partially Observable ...
3
votes
0answers
18 views

Is there any way of generating fixed-length sequences with RNNs?

Is there any way of generating fixed-length sequences with RNNs? I want to tell my character level RNN to generate a name of length 3, 4, 5 and so on. I haven't found anything online like this, but my ...
1
vote
1answer
13 views

how to produce documents like factset blackline?

factset blackline reports essentially can compare two 10-Q SEC filings and show you the difference between the two documents. It highlights added items in green and removed items in red + ...
3
votes
0answers
28 views

How to predict an event (or action) based on a window of time-series measurements?

I have an input vector $X$, which contains a series of measurements within a period, e.g. 100 measurements in 1 sec. The goal is to predict an event, let's say, moving forward, backward or static. I ...