Questions tagged [recurrent-neural-networks]

For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Understanding graphs of the mean square error: relationships between val loss and train loss

I am currently working with some models aimed at predicting time series (89 days for training, 22 for testing), including a CNN LSTM and a convLSTM. When training these models, I had the following ...
0
votes
0answers
9 views

Finding Cycles in a State Sequence

Suppose I observe a set of states $\mathbf{X} = \{X_{1}, X_{2}, \ldots, X_{K}\}$ over time. I assume that there exist $M$ cycles $\mathbf{C} = \{C_{1}, C_{2}, \ldots, C_{M} \}$ in the observed state ...
0
votes
2answers
39 views

Is there a neural network that accepts both the current input and previous output?

I am quite new to neural networks. I am trying to implement in Python a neural network having only one hidden layer with $N$ neurons and $1$ output layer. The point is that I am analyzing time series ...
1
vote
0answers
14 views

Keras model accuracy not improving beyond threshold

I am currently working on a public project for the National Weather Model. We are experimenting with using a recurrent neural network to replace the output of a quadratic formula that is in use. The ...
-1
votes
0answers
11 views

Decoder.predict not working inside a training model

Hi I wanted to discuss a certain situation I have.. I have a LSTM code and I want to feed in an encoded sequence to it.. After that I want to run a decoder and classifier on the output it gives while ...
-1
votes
0answers
25 views

Training a Neural Network with Hexadecimal input

I currently started working in the field of Machine Learning and have been stuck on a problem where I have to train a Neural Network with a dataset containing hexadecimal inputs. I found that we can ...
0
votes
0answers
19 views

What are some solutions for dealing with time series data that are recorded at uneven intervals?

Let's say I have a time series data which is a bunch of observations that occur at different time stamps and intervals. For example, my observations come from a camera located at a traffic ...
0
votes
0answers
23 views

How to find the derivative of a dynamic neuron model, which depends on previous states of the neuron?

This is the equation where n denotes the current state, (n-1) denotes the state in the previous step etc. And to do back-propagation I need to find partial derivatives over each of the variables. For ...
2
votes
1answer
59 views

How to use text as an input for a neural network - regression problem? How many likes/claps an article will get

I am trying to predict the number of likes an article or a post will get using a NN. I have a dataframe with ~70,000 rows and 2 columns: "text" (predictor - strings of text) and "likes&...
1
vote
1answer
46 views

Recommendations or resources for neural network/deep learning for time series application?

I know there are quite a few good deep learning books out there, but most explain neural networks and deep learning via application on images. If there are examples/code, they are often done on the ...
3
votes
1answer
110 views

What exactly are the “parameters” in GPT-3's 175 billion parameters and how are they chosen/generated?

When I studied neural networks, parameters were learning rate, batch size etc. But even GPT3's ArXiv paper does not mention anything about what exactly the parameters are, but gives a small hint that ...
1
vote
0answers
35 views

How can one be sure that a particular neural network architecture would work?

Traditionally, when working with tabular data, one can be sure(or at least know) that a model works because the included features could explain a target variable, say "Price of a ticket" ...
0
votes
1answer
37 views

What is the big fuzz about SHA-RNN versus Transformers?

In his paper introducing SHA-RNN (https://arxiv.org/pdf/1911.11423.pdf) Stephen Merity states that neglecting one direction of research (in this case LSTMs) over another (transformers) merily because ...
0
votes
1answer
47 views

Input tensor shape order for RNN (PyTorch)

I am confused as to why the sequence length is the first dimension of the input tensor for an RNN when batch size is the first dimension for any other kind of network (Linear/CNN/etc.). This makes me ...
1
vote
2answers
62 views

Why the cost/loss starts to increase for some iterations during the training phase?

I am trying to build a recurrent neural network from scratch. It's a very simple model. I am trying to train it to predict two words (dogs and gods). While training, the value of cost function starts ...
0
votes
0answers
21 views

How to feed the LSTM with different length for the latest time step?

I am having a training data set for a time-series dataset like below: ...
2
votes
1answer
57 views

Why are RNNs used in some computer vision problems?

I am learning computer vision. When I was going through implementations of various computer vision projects, some OCR problems used GRU or LSTM, while some did not. I understand that RNNs are used ...
0
votes
0answers
41 views

What does the notation “for t=T to 1,−1 do” in terms of time steps, in deep recurrent q network?

In looking at an algorithm in the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning. Here is the full algorithm: What does the notation ...
1
vote
0answers
31 views

What is the role of embeddings in a deep recurrent Q network?

When describing the model architecture for a deep recurrent q network, the authors of the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning each agent consists of a recurrent ...
1
vote
1answer
41 views

Why does my “entropy generation” RNN do so badly?

I'm new to relatively RNNs, and I'm trying to train generative and guessing neural networks to produce sequences of real numbers that look random. My architecture looks like this (each "circle&...
2
votes
1answer
58 views

What is the difference between LSTM and fully connected LSTM?

I'm currently trying to understand the difference between a vanilla LSTM and a fully connected LSTM. In a paper I'm reading, the FC-LSTM gets introduced as FC-LSTM may be seen as a multivariate ...
1
vote
0answers
40 views

Do you have to add a dense layer onto the final layer of an LSTM?

If my understanding of an LSTM is correct then the output from each LSTM unit is the hidden state from that layer. For the final layer if I wanted to predict e.g. a scalar real number, would I want to ...
0
votes
0answers
35 views

Why in RL function approximators with recurrent structures can learn planning?

In the paper An Investigation of Model-Free Planning the authors use ConvLSTM to learn a planning function. In particular, for each input x_t at time-step ...
0
votes
0answers
22 views

Neural Network architecture for going from scalar input to time series outputs?

I have a problem where I know p features which are each scalar values and the output of 1 set of those features is a time history. Is there a specific neural network (NN) type architecture that can ...
2
votes
1answer
33 views

How do LSTM or GRU gates learn to specialize in their desired tasks?

While I was studying the equations for the computation inside GRU and LSTM units, I realized that although the different gates have different Weight matrices, their overall structure is the same. They ...
3
votes
1answer
82 views

What are the keys and values of the attention model for the encoder and decoder in the “Attention Is All You Need” paper?

I have recently encountered the paper on NLP. It is very new to me and I am still unable to see how that works. I have used all the resources over there from the original paper to Youtube videos and ...
2
votes
0answers
35 views

How to understand the matrices used in the Attention layer?

Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
2
votes
0answers
31 views

Why do we need recurrent neural networks instead of feed-forward neural networks? [duplicate]

Why do we need recurrent neural networks instead of feed-forward neural networks? What are the advantages of RNNs compared with FFNNs?
1
vote
1answer
25 views

How to exclude sections of bad data from time-series data before training an LSTM network

I am using LSTM network for predicting IOT time-series data receiving from un-reliable devices and networks. This results in several multiple sections [continuous streak of bad data for several days ...
1
vote
1answer
37 views

How to make a LSTM network to predict sequence only after input sequence is finished?

I am learning to use a LSTM model to predict time series data. Specifically, I hope the network should output a sequence (with multiple time steps) only after the input sequence has finished feeding ...
1
vote
0answers
27 views

Deep learning techniques with time-fixed, time-dependent and imaging data

I have a question about the use of deep learning techniques with time-fixed features and images (setting 1) and time-dependent features (setting 2). (I am pretty new to the deep learning world so ...
1
vote
1answer
67 views

Why do we use a delay when feeding our input data to the echo state network?

I'm new to working with neural networks and have recently began implementing neural networks for time series forecasting in some of my work. I've been particularly using Echo State Networks and have ...
3
votes
0answers
41 views

How does backpropagation work in LSTMs?

After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in ...
1
vote
0answers
21 views

Is the number of bidirectional LSTMs in seq2seq model equal to the maximum length of input text/characters?

I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/. It ...
1
vote
0answers
26 views

My LSTM text classification model seems not learn anything in early epochs

I am trying to use LSTM to do text classification and monitor the training process with tensorboard. But it seems that this model doesn't learn anything in early epochs. Is it normal for LSTM networks?...
2
votes
0answers
29 views

Can you use transformer models to do autocomplete tasks?

I've researched online and seen many papers on the use of RNNs (like LSTMs or GRUs) to autocomplete for, say, a search engine, character by character. Which makes sense since it inherently predicts ...
1
vote
0answers
17 views

How do CNNs or RNNs “stack the feature of nodes by a specific order”?

I am trying to understand the following statement taken from the paper Graph Neural Networks: A Review of Methods and Applications (2019). Standard neural networks like CNNs and RNNs cannot handle ...
1
vote
0answers
15 views

How many spectrogram frames per input character does text-to-speech (TTS) system Tacotron-2 generate?

I've been reading on Tacotron-2, a text-to-speech system, that generates speech just-like humans (indistinguishable from humans) using the GitHub https://github.com/Rayhane-mamah/Tacotron-2. I'm very ...
2
votes
0answers
21 views

Reservoir of LSM vs. FF-NN or ELM

The reservoir of the Liquid State Machine is an array of random numbers connected to each other with a probability depending on the distance between each other. Because of this connection with each ...
2
votes
2answers
67 views

How can Siamese Networks be viewed as RNNs?

"Single-object tracking commonly uses Siamese networks, which can be seen as an RNN unrolled over two time-steps." (from the SQAIR paper) I'm wondering how Siamese networks can be viewed as RNNs, as ...
0
votes
0answers
28 views

RNN classifying targets based on size~

I have implemented a recurrent neural network in keras with the following architecture: custom clustering layers from this tutorial: https://www.dlology.com/blog/how-to-do-unsupervised-clustering-...
2
votes
1answer
83 views

What are modern state-of-the-art solutions in prediction of time-series?

I wanted to ask you about the newest achievements in time series analysis (mostly prediction). What state-of-the-art solutions (as in frameworks, papers, related projects) do you know that can be used ...
2
votes
0answers
33 views

Why Pixel RNN (Row LSTM) can capture triangular contexts?

I'm reading the paper Pixel Recurrent Neural Network. I have a question about Row LSTM. Why Row LSTM can capture triangular contexts? In this paper, the kernel of the one-dimensional convolution ...
2
votes
1answer
38 views

Neural networks with internal dynamics in the state-space form

Neural networks with feedback (Hopfield, Hamming, etc.) differ from ordinary neural networks (multilayer perceptrons, etc.), which turns them into a dynamic element with its own internal dynamics (if ...
2
votes
0answers
70 views

What is the time complexity of the forward pass and back-propagation of the sequence-to-sequence model with and without attention?

I keep looking through the literature, but can't seem to find any information regarding the time complexity of the forward pass and back-propagation of the sequence-to-sequence RNN encoder-decoder ...
1
vote
0answers
24 views

How to derive asymptotic run-time of seq2seq RNN? [duplicate]

I want to know the computational complexity of a sequence to sequence RNN in terms of the input and output length. I was wondering how I might go about deriving this in both training and inference.
1
vote
1answer
118 views

How to train a LSTM with multidimensional data

I am trying to train a LSTM, but I have some problems regarding the data representation and feeding it into the model. My data is a numpy array of three dimensions: One sample consist of a 2D matrix ...
11
votes
3answers
3k views

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I am reading the article How Transformers Work where the author writes Another problem with RNNs, and LSTMs, is that it’s hard to parallelize the work for processing sentences, since you have to ...
2
votes
1answer
35 views

Is this a correct visual representation of a recurrent neural network (RNN)?

This is a picture of a recurrent neural network (RNN) found on a udemy course (Deep Learning A-Z). The axis at the bottom is "time". In a time series problem, each yellow row from left to right ...
1
vote
1answer
22 views

Inner working of Bidirectional RNNs

I'm trying to understand how Bidirectional RNNs work. Specifically, I want to know whether a single cell is used with different states, or two different cells are used, each having independent ...

1
2 3 4 5