Questions tagged [recurrent-neural-networks]

For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

How to Create a Fixed-Length, Binary, Sequence of Tokens Embedding?

Say I have 10 classes represented by 1 x n_classes vector of binary My goal is to embed a sequence of 1xN binary so that I could also model the class-co occurrence. Say, class A, B, D are present and ...
0 votes
0 answers
19 views

Why remove stop words, numbers in a conversational chatbot?

I have been working on a conversational chatbot recently using movie dialogues corpus dataset, since i am very new to this i started to see if there's already code available for chatbots. I came ...
0 votes
0 answers
13 views

Surrogate model to produce time series from parameter set

Say I have a model $M$ that takes in a parameter vector $\beta$, and produces a (numerical) time series. This could be a complicated model (e.g. a bespoke enzyme reaction model), or something simple ...
  • 1
2 votes
1 answer
73 views

Clarification on GANs for text generation

A GAN-like architecture for text generation is proposed in 'Generative Adversarial Networks for Text Generation'. The setup is the following: The generator of the GAN is proposed to be a recurrent ...
0 votes
0 answers
10 views

Predicting time spent to build a metal piece using RNN

My data consists in many metal pieces which are put together to make a final metal mould. To make each of this metal pieces, machinery recieves many operations to follow, like chopping, facing, etc... ...
  • 1
1 vote
0 answers
23 views

What are Reservoir computers used for today?

Reservoir computers were very popular in the early 2000s. From what I understand, the advantage of reservoir computers is that, as opposed to generic recurrent neural networks, training is only done ...
0 votes
0 answers
13 views

How to include input features in Recurrent Neural Networks

I want to predict a time series. I want to use methods like Recurrent Neural Networks (RNN) but I want to also have some other input features. I mean as far as I know RNN predicts the future just ...
  • 141
0 votes
1 answer
18 views

Why do smaller weights converge faster for RNNs?

I am writing a Recurrent Neural Network using only the NumPy library for a binary classification problem. When I initialize the weights with np.random.randn, after 1000 epochs it gets ~60% accuracy, ...
  • 15
0 votes
0 answers
43 views

How is it possible to use batches of data from within the same sequence with an LSTM?

ETA: More concise wording: Why do some implementations use batches of data taken from within the same sequence? Does this not make the cell state useless? Using the example of an LSTM, it has a hidden ...
  • 1,266
0 votes
0 answers
12 views

Tangent/slope at a point of a recurrent neural network

I am using a recurrent neural network for data of the form $\{(x_t, y_t)\}_{t=1}^T$. I defined the input of the RNN as a sequence $(x_{t-1}, x_t, y_{t-1})$ and output as $y_t$. My RNN has therefore ...
0 votes
2 answers
51 views

How to define a loss function for multi-label problem?

I have voice recordings which are labelled by not only a single label but multiple labels. Each voice recording corresponds to one of class labels within a set. In other words, the training instance ...
  • 103
0 votes
1 answer
44 views

How does the hidden activation differ from the output, at any time step for a SimpleRNN?

I am watching the Sequence models course taught by Andrew Ng. I am a bit lost on the SimpleRNN lecture. As per the lecture, at each time step, there's an output y from a hidden layer and an input ...
1 vote
1 answer
46 views

Is it possible for AlphaGo Zero to use recurrent networks to achieve similar performance?

AlphaGo Zero stacks 7 board history along with the current board together to form the input to the network. However, is it possible to use an RNN to replace the input of history and achieve similar ...
  • 193
1 vote
0 answers
92 views

Why can't I reproduce my results in keras using random seed? [closed]

I was doing a task using RNN to predict a time series movement. I want to make my results reproducible. So I strictly followed this post: https://stackoverflow.com/questions/32419510/how-to-get-...
0 votes
1 answer
40 views

What model can solve vector to vector prediction?

I am totally newbie into serial prediction. I am think about which model or AI paradigm can be used to do vector to vector prediction? For instance, [1,0,1] ^ [0,1,0] = [1,1,1] Another example could ...
  • 111
0 votes
0 answers
19 views

Finding "look_back" & "look_ahead" hyper-parameters for Seq2Seq models

For Seq2Seq deep learning architectures, viz., LSTM/GRU and multivariate, multistep time series forecasting, it is important to convert the data to a 3D dimension: (batch_size, look_back, ...
  • 205
2 votes
0 answers
150 views

Why use a fully connected layer for attention?

In the paper Neural Machine Translation by Jointly Learning to Align and Translate, attention is used with a single fully connected layer. Specifically, in the auto-regressive set up (equation 4), the ...
  • 121
0 votes
1 answer
121 views

How should you reshape data before feeding it to LSTM layers?

I was curious if anyone had any advice on how to reshape data for a recurrent neural network. What I've been doing is array.reshape(len(X_train), # of points in time, # of features) And then in the ...
0 votes
0 answers
45 views

Predicting using time-series data and static data?

I have recently been working on predicting the final value of articles on Steemit.com using downloaded data. I have a large variety of features which divide into two types. Features which change over ...
0 votes
0 answers
64 views

Why are separate, bigger Encoder-Decoder architectures used instead of Bidirectional RNNs/Transformers for Seq2Seq tasks?

Whether with RNNs or Transformers, Encoder-Decoder networks are used for Sequence to Sequence (Seq2Seq) tasks, like Machine Translation. Why are separate, bigger Encoder-Decoder networks used for this ...
0 votes
0 answers
14 views

Vector to sequence RNNs: do they take a random initial "prompt"?

I am going through the Deep Learning book by Ian Goodfellow (here) and came by the architecture for a vector to sequence RNN (Figure 10.9). I am not sure I understand how this architecture works and ...
0 votes
0 answers
18 views

Using an RNN for predicting columns of characters

I'm making an RNN using pytorch to learn from columns of tiles (each tile represented by a text character) and predict the next column of tiles. The training sequences are from maps of level data ...
1 vote
0 answers
30 views

When training a seq2seq model is it better to train using the models outputs or expected outputs?

When training any seq2seq model you have a target and a source. The source may be a sentence ...
  • 1,266
0 votes
0 answers
37 views

How are session-parallel mini-batches used for training RNNs for session-based recommender tasks?

I am reading this paper on session-based recommenders with RNNs: https://arxiv.org/abs/1511.06939. During the training phase, the authors apply what they call "session-parallel mini-batches,"...
0 votes
1 answer
674 views

What is the difference between CNN-LSTM and RNN?

I'm starting to study RNN for a project of video prediction, but I encounter these CNN-LSTM models. Initially, I thought that is another name for RNN, but I think I get it wrong. Since I'm a beginner ...
0 votes
0 answers
187 views

Is attention always better then an RNN/CNN?

We've all read the attention is all you need paper, but is it really all you need? Can you effectively replace any RNN/CNN with an attention transformer and see better results?
  • 1,266
0 votes
0 answers
60 views

What is an attractor network?

Surprisingly, this wasn't asked before - at least there was one related question without any answers What is a continuous-attractor neural network?. So, what is an attractor network, and why should ...
  • 111
1 vote
0 answers
14 views

Applications of one-to-one recurrent networks

I'm studying recurrent neural networks. Reading this page where it lists different types of recurrent network architectures, I think think of applications involving one-to-many (speech/sentence ...
  • 111
0 votes
0 answers
30 views

What could be causing the poor performance (MSE) of a dense neural network on a real time-series dataset?

I am trying to understand time series analysis and actually I am following the course "Sequences, Time Series and Prediction" in Coursera. The course is based on a synthetic dataset, ...
0 votes
1 answer
38 views

How can my RNN get way better results than my ANN [closed]

So, I'm using the same dataset in both models but my RNN gets a 95% accuracy and my ANN gets 52%. It is a time series, binary classification problem, and I know that RNN is better than ANN for time ...
0 votes
0 answers
32 views

Why doesn't torch use a nonlinearity in its RNN implementation?

The RNN example implementation and the RNN tutorial from pytorch doesn't use a nonlinarity in the hidden layer. Shouldn't the network have at least one nonlinear activation to be able to learn ...
  • 240
0 votes
0 answers
46 views

MLP or RNN for Regression of Smooth Function (No Time Data)?

My Problem consists of Input sequences in the form of $x=[B,z]$ and one output $y_i$ for each data point $x_i=[B,z_i]$. For one sequence/dataset $B$ is a constant, whereas $z$ is continously between 0 ...
  • 41
0 votes
0 answers
52 views

LSTM: Simple value series vs Complex value series

A model for learning a trend graph can be this way: To learn a sequence of N numbers LSTM layer of M units Dense output node of 1 unit The problem is a trend graph in history can be simple: Case 1:...
  • 1,223
1 vote
2 answers
45 views

Is there any reason for giving an index to a token based on its frequency in the text?

In pre-processing of text, we need to assign a number for each token in a text. Then only we can pass it to a model. In pre-processing of text, we need to assign a number for each token in a text. The ...
  • 3,251
0 votes
0 answers
36 views

Does it make sense to compare images (samples) with words (features)?

Consider the following paragraphs from the introduction of the chapter named Recurrent Neural Networks from the textbook titled Dive into Deep Learning So far we encountered two types of data: ...
  • 3,251
0 votes
0 answers
293 views

Next Sentence Prediction for 5 sentences using BERT

I am given a dataset in which each instance consisting of 5 sentences. The goal is to predict the sequence of numbers which represent the order of these sentences. For example, given a story: He went ...
1 vote
0 answers
47 views

neat - what is the purpose of looped networks?

So im writing my own implementation of NEAT and i'm wondering how looped networks (like one shown in the image) can be useful. I'll probably implement them anyway because i want to fiddle around with ...
  • 11
1 vote
0 answers
33 views

Are RNN a good approach to solve this type of problem?

I have a problem that can be optimized by taking five actions, and finally, after a series of steps to achieve a solution. The actions (1 to 5) are picked randomly. A time-step (epoch) is concluded ...
  • 111
0 votes
0 answers
53 views

How to reduce the variance of stochastic policy gradient for continuous actions in a partially observable environment?

I am trying to implement a stochastic policy gradient for continuous actions in a partially observable CartPole environment. Specifically, only the current cart position and pole angle are visible, ...
  • 193
-2 votes
1 answer
66 views

My accuracy wont improve in tensorflow [closed]

I've been trying to figure out why this model won't train (the accuracy stays at 0). ...
  • 125
0 votes
0 answers
32 views

How does Stack-Augmented Recurrent Nets in work?

I am new to RNN/LSTM and I am working on a project about language modeling. I just got familiarized with simple RNN and LSTM. However, these simple models did not achieve the performance I want. Since ...
0 votes
0 answers
26 views

Is smoothing wrong in temporal predictions?

I found this paper from 2003 about predicting Forex rates: Using Recurrent Neural Networks To Forecasting of Forex. At the end of page 11, they say The network we built had two inputs and one output. ...
0 votes
1 answer
81 views

Is my intuition about RNN wrong?

Until today, my intuition about RNN (LSTM/GRU) was that this is some kind of NN that can remember previous inputs. Consider a task where you need to predict 0 if the previous input was 1. For example: ...
  • 109
1 vote
0 answers
19 views

What does "These designs employ skip connections to avoid a situation where the shortest path between time steps increases" mean?

Less popular alternatives include adding layers to the connections from input to the hidden state, between hidden states, or from the hidden state to the output. These designs employ skip connections ...
0 votes
0 answers
27 views

How to train with a non-differentiable activation function (such as SVT in deep unrolling low-rank optimization)?

I planned to design a deep unfolding for decomposition into low-rank and sparse in Pytorch environment. I read this paper that might help me to understand how to do it. I always taught that this model ...
0 votes
1 answer
59 views

Order of operations on sparse recurrent network alters the output. How to deal with it?

I'm working on an implementation of NEAT, which evolves neural networks with small and sparse topologies. Evaluating a sparse and possibly recurrent network requires a different approach than the ...
0 votes
0 answers
29 views

Could anyone please explain this sentence about training in parallel?

One way to reduce the computational complexity of hidden state recurrences is to connect a unit's hidden state to the prior unit's output rather than its hidden state. The resulting RNN has a lower ...
0 votes
0 answers
13 views

What is the primary advantage of viewing RNN as a directed graphical model?

While reading the chapter titled "Sequence Modeling: Recurrent and Recursive Nets" from the textbook named Deep Learning by Ian Goodfellow et al, I came across a subsection 10.2.3 titled &...
  • 3,251
1 vote
1 answer
49 views

How to do testing for an RNN that was trained with teacher forcing only?

If an RNN is trained using only the teacher forcing, then the network takes the actual output from the previous time step as input to the hidden state the next time step. We know that the actual ...
  • 3,251
0 votes
0 answers
23 views

Which kind of RNNs are mostly used in applications: hidden-hidden or actual output-hidden?

I came across two types of RNN while reading the chapter titled Sequence Modeling: Recurrent and Recursive Nets of the textbook named Deep Learning by Ian Goodfellow et al. First type: Recurrent ...
  • 3,251

1
2 3 4 5
7