2022 Developer Survey is open! Take survey.

Questions tagged [recurrent-neural-networks]

For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

Predicting using time-series data and static data?

I have recently been working on predicting the final value of articles on Steemit.com using downloaded data. I have a large variety of features which divide into two types. Features which change over ...
user avatar
0 votes
0 answers
20 views

Why are separate, bigger Encoder-Decoder architectures used instead of Bidirectional RNNs/Transformers for Seq2Seq tasks?

Whether with RNNs or Transformers, Encoder-Decoder networks are used for Sequence to Sequence (Seq2Seq) tasks, like Machine Translation. Why are separate, bigger Encoder-Decoder networks used for this ...
user avatar
0 votes
0 answers
11 views

Vector to sequence RNNs: do they take a random initial "prompt"?

I am going through the Deep Learning book by Ian Goodfellow (here) and came by the architecture for a vector to sequence RNN (Figure 10.9). I am not sure I understand how this architecture works and ...
user avatar
0 votes
0 answers
13 views

Using an RNN for predicting columns of characters

I'm making an RNN using pytorch to learn from columns of tiles (each tile represented by a text character) and predict the next column of tiles. The training sequences are from maps of level data ...
user avatar
1 vote
0 answers
13 views

When training a seq2seq model is it better to train using the models outputs or expected outputs?

When training any seq2seq model you have a target and a source. The source may be a sentence ...
user avatar
  • 1,324
0 votes
0 answers
15 views

How are session-parallel mini-batches used for training RNNs for session-based recommender tasks?

I am reading this paper on session-based recommenders with RNNs: https://arxiv.org/abs/1511.06939. During the training phase, the authors apply what they call "session-parallel mini-batches,"...
user avatar
0 votes
1 answer
49 views

What is the difference between CNN-LSTM and RNN?

I'm starting to study RNN for a project of video prediction, but I encounter these CNN-LSTM models. Initially, I thought that is another name for RNN, but I think I get it wrong. Since I'm a beginner ...
user avatar
0 votes
0 answers
30 views

Is attention always better then an RNN/CNN?

We've all read the attention is all you need paper, but is it really all you need? Can you effectively replace any RNN/CNN with an attention transformer and see better results?
user avatar
  • 1,324
0 votes
0 answers
50 views

What is an attractor network?

Surprisingly, this wasn't asked before - at least there was one related question without any answers What is a continuous-attractor neural network?. So, what is an attractor network, and why should ...
user avatar
  • 101
1 vote
0 answers
13 views

Applications of one-to-one recurrent networks

I'm studying recurrent neural networks. Reading this page where it lists different types of recurrent network architectures, I think think of applications involving one-to-many (speech/sentence ...
user avatar
  • 111
0 votes
0 answers
24 views

What could be causing the poor performance (MSE) of a dense neural network on a real time-series dataset?

I am trying to understand time series analysis and actually I am following the course "Sequences, Time Series and Prediction" in Coursera. The course is based on a synthetic dataset, ...
user avatar
0 votes
1 answer
36 views

How can my RNN get way better results than my ANN [closed]

So, I'm using the same dataset in both models but my RNN gets a 95% accuracy and my ANN gets 52%. It is a time series, binary classification problem, and I know that RNN is better than ANN for time ...
user avatar
0 votes
0 answers
15 views

Why doesn't torch use a nonlinearity in its RNN implementation?

The RNN example implementation and the RNN tutorial from pytorch doesn't use a nonlinarity in the hidden layer. Shouldn't the network have at least one nonlinear activation to be able to learn ...
user avatar
  • 240
0 votes
0 answers
40 views

MLP or RNN for Regression of Smooth Function (No Time Data)?

My Problem consists of Input sequences in the form of $x=[B,z]$ and one output $y_i$ for each data point $x_i=[B,z_i]$. For one sequence/dataset $B$ is a constant, whereas $z$ is continously between 0 ...
user avatar
  • 41
0 votes
0 answers
40 views

LSTM: Simple value series vs Complex value series

A model for learning a trend graph can be this way: To learn a sequence of N numbers LSTM layer of M units Dense output node of 1 unit The problem is a trend graph in history can be simple: Case 1:...
user avatar
  • 1,153
1 vote
2 answers
33 views

Is there any reason for giving an index to a token based on its frequency in the text?

In pre-processing of text, we need to assign a number for each token in a text. Then only we can pass it to a model. In pre-processing of text, we need to assign a number for each token in a text. The ...
user avatar
  • 3,029
0 votes
0 answers
33 views

Does it make sense to compare images (samples) with words (features)?

Consider the following paragraphs from the introduction of the chapter named Recurrent Neural Networks from the textbook titled Dive into Deep Learning So far we encountered two types of data: ...
user avatar
  • 3,029
0 votes
0 answers
59 views

Next Sentence Prediction for 5 sentences using BERT

I am given a dataset in which each instance consisting of 5 sentences. The goal is to predict the sequence of numbers which represent the order of these sentences. For example, given a story: He went ...
user avatar
1 vote
0 answers
30 views

neat - what is the purpose of looped networks?

So im writing my own implementation of NEAT and i'm wondering how looped networks (like one shown in the image) can be useful. I'll probably implement them anyway because i want to fiddle around with ...
user avatar
  • 11
1 vote
0 answers
31 views

Are RNN a good approach to solve this type of problem?

I have a problem that can be optimized by taking five actions, and finally, after a series of steps to achieve a solution. The actions (1 to 5) are picked randomly. A time-step (epoch) is concluded ...
user avatar
  • 111
0 votes
0 answers
18 views

How to reduce the variance of stochastic policy gradient for continuous actions in a partially observable environment?

I am trying to implement a stochastic policy gradient for continuous actions in a partially observable CartPole environment. Specifically, only the current cart position and pole angle are visible, ...
user avatar
  • 173
-2 votes
1 answer
49 views

My accuracy wont improve in tensorflow [closed]

I've been trying to figure out why this model won't train (the accuracy stays at 0). ...
user avatar
  • 125
0 votes
0 answers
20 views

How does Stack-Augmented Recurrent Nets in work?

I am new to RNN/LSTM and I am working on a project about language modeling. I just got familiarized with simple RNN and LSTM. However, these simple models did not achieve the performance I want. Since ...
user avatar
0 votes
0 answers
23 views

Is smoothing wrong in temporal predictions?

I found this paper from 2003 about predicting Forex rates: Using Recurrent Neural Networks To Forecasting of Forex. At the end of page 11, they say The network we built had two inputs and one output. ...
user avatar
0 votes
1 answer
59 views

Is my intuition about RNN wrong?

Until today, my intuition about RNN (LSTM/GRU) was that this is some kind of NN that can remember previous inputs. Consider a task where you need to predict 0 if the previous input was 1. For example: ...
user avatar
  • 109
1 vote
0 answers
17 views

What does "These designs employ skip connections to avoid a situation where the shortest path between time steps increases" mean?

Less popular alternatives include adding layers to the connections from input to the hidden state, between hidden states, or from the hidden state to the output. These designs employ skip connections ...
user avatar
0 votes
0 answers
18 views

How to train with a non-differentiable activation function (such as SVT in deep unrolling low-rank optimization)?

I planned to design a deep unfolding for decomposition into low-rank and sparse in Pytorch environment. I read this paper that might help me to understand how to do it. I always taught that this model ...
user avatar
0 votes
1 answer
40 views

Order of operations on sparse recurrent network alters the output. How to deal with it?

I'm working on an implementation of NEAT, which evolves neural networks with small and sparse topologies. Evaluating a sparse and possibly recurrent network requires a different approach than the ...
user avatar
0 votes
0 answers
29 views

Could anyone please explain this sentence about training in parallel?

One way to reduce the computational complexity of hidden state recurrences is to connect a unit's hidden state to the prior unit's output rather than its hidden state. The resulting RNN has a lower ...
user avatar
0 votes
0 answers
12 views

What is the primary advantage of viewing RNN as a directed graphical model?

While reading the chapter titled "Sequence Modeling: Recurrent and Recursive Nets" from the textbook named Deep Learning by Ian Goodfellow et al, I came across a subsection 10.2.3 titled &...
user avatar
  • 3,029
1 vote
1 answer
33 views

How to do testing for an RNN that was trained with teacher forcing only?

If an RNN is trained using only the teacher forcing, then the network takes the actual output from the previous time step as input to the hidden state the next time step. We know that the actual ...
user avatar
  • 3,029
0 votes
0 answers
22 views

Which kind of RNNs are mostly used in applications: hidden-hidden or actual output-hidden?

I came across two types of RNN while reading the chapter titled Sequence Modeling: Recurrent and Recursive Nets of the textbook named Deep Learning by Ian Goodfellow et al. First type: Recurrent ...
user avatar
  • 3,029
1 vote
0 answers
25 views

Can teacher forcing in RNN ensure Turing completeness?

RNN has the same capability as a universal Turing machine. But I am confused whether RNN holds the same capabilities if we use teacher forcing. Consider the following excerpts from paragraphs taken ...
user avatar
  • 3,029
1 vote
0 answers
48 views

Is the capability of RNN more than the capability of MLP?

Consider the following excerpt paragraph taken from the section titled "Recurrent Neural Networks" of the chapter 10: Sequence Modeling: Recurrent and Recursive Nets of the textbook named ...
user avatar
  • 3,029
4 votes
1 answer
79 views

Is there any relation between the recursive neural network and recurrent neural network?

Recurrent neural networks, abbreviated as RNNs, are widely used in deep learning literature, especially for text processing. Are they related to recursive neural networks in any way? I am asking for ...
user avatar
  • 3,029
1 vote
1 answer
37 views

When does an RNN use the connections that help in going backward in time?

Consider the following paragraph taken from chapter 10: Sequence Modeling: Recurrent and Recursive Nets of the textbook named Deep Learning by Ian Goodfellow et al mentioning the connections of RNN to ...
user avatar
  • 3,029
0 votes
0 answers
25 views

What does "statistical strength" mean in this context?

Consider the following excerpt from a paragraph taken from chapter 10: Sequence Modeling: Recurrent and Recursive Nets of the textbook named Deep Learning by Ian Goodfellow et al regarding the ...
user avatar
  • 3,029
0 votes
1 answer
62 views

Is a recurrent layer same as LSTM or single-layered LSTM?

In MLP, there are neurons that form a layer. Each hidden layer gives a vector of number that is the output of that layer. In CNN, there are kernels that form a convolutional layer. Each layer gives ...
user avatar
  • 3,029
0 votes
0 answers
32 views

What is all necessary types of data for a bidirectional RNN to learn embeddings?

Bidirectional RNNs are used for generating the semantic vectors of the text at the sentence level and word level. In order to train a CNN for the classification tasks, images, and labels/outputs are ...
user avatar
  • 3,029
0 votes
0 answers
26 views

What are the types of inputs used for RNN in literature given sentences?

Suppose there are $m$ sentences in a text file and the number of distinct words is equal to $n$. The goal is to get word embeddings using RNN. We know that it is impossible to pass any word, which is ...
user avatar
  • 3,029
1 vote
1 answer
36 views

Is image machine translation done in two steps?

Suppose I have images of hand-written Japanese text. If I want to translate those images, would my ML algorithm be a 2-step model (for example, a CNN to convert the image into Japanese characters/...
user avatar
1 vote
1 answer
139 views

What exactly is embedding layer used in RNN encoders?

I am reading about RNN encoders. I came across the following line from this code. And I am facing difficulty in understanding the theoretical details regarding it. ...
user avatar
  • 3,029
0 votes
1 answer
42 views

What is a "mask" in the context o RNN-based encoders?

While reading source code related to RNN encoders, I've come across the term mask as input to the encoder. What exactly is it?
user avatar
  • 3,029
1 vote
1 answer
21 views

Writing a loss function for "how far can this output be pushed"

I'm trying to train a function for a industrial-process-control-like system. This is my first attempt at a custom training, so feel free to point out any invalid assumptions. I've got one input and ...
user avatar
  • 111
1 vote
1 answer
34 views

What kind of NN to use to find misprints in test

I have a bunch of unique full names of users. I made pseudo-physical model to emulate misprints of desktop and mobile users (hence, fatfingering, jumpy fingers, accidentals touches of touch bar etc.) ...
user avatar
0 votes
1 answer
32 views

Neural network for recognizing ship types based on location series

I am building a neural network for recognizing ship types based on a 1000-long series of location data (latitude-longitude, normalized to account for different km/longitude° metrics, so that vector ...
user avatar
0 votes
1 answer
33 views

What is remembering in Hopfield network?

Hopfield is a simple and traditional network. We feed into the network some patterns (Learning/Training). Actually, there is no training in Hopfield as the weight calculation is just adding up all the ...
user avatar
  • 101
3 votes
1 answer
109 views

In a Temporal Convolutional Network, how is the receptive field different from the input size?

I'm playing around with TCN's lately and I don't understand one thing. How is the receptive field different from the input size? I think that the receptive field is the time window that TCN considers ...
user avatar
1 vote
1 answer
27 views

Does Seq2Seq decoder take a special vector or the weights of the last encoder cell as an output?

I'm reading Sequence to Sequence Learning with Neural Networks and there's a thing that I couldn't quite grasp. Paper says the encoder outputs a vector to be fed to the decoder. More precisely Our ...
user avatar
  • 145
0 votes
0 answers
63 views

What would be the total number of learnable parameters of the RNN encoder of this encoder-decoder architecture for machine translation?

Here's a quiz. My answer is different from the teacher's, so I'm wondering what answer would you pick up. We use a sequence-to-sequence (encoder-decoder) system to perform machine translation. We ...
user avatar
  • 113

1
2 3 4 5
7