Questions tagged [recurrent-neural-networks]

For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.

Filter by
Sorted by
Tagged with
3
votes
1answer
38 views

Is there a common way to build a neural network that seeks to extract spatial and temporal information simultaneously?

Is there a common way to build a neural network that seeks to extract spatial and temporal information simultaneously? Is there an agreed up protocol on how to extract this information? What ...
1
vote
1answer
23 views

How does vanish gradient restrict RNN to not work for long range dependencies?

I am really trying to understand deep learning models like RNN, LSTMs etc. I have gone through many tutorials of RNN and have learned that RNN cannot work for long Range dependencies, like: Consider ...
-1
votes
0answers
31 views

How to properly build a neural net of a physics simulation [closed]

I have a rocket simulation that produces 4 inputs and 4 outputs. My plan is to run 5000 simulation runs and disperse the 4 inputs then write the 4 outputs to a CSV file. I will then use TensorFlow and ...
1
vote
1answer
13 views

How do LSTMs work if the following two matrices are not able to be multiplied?

In the above diagram, the shape of some of the matrices can be seen in the yellow highlight. For instance: The hidden state at timestep t-1 ($h_{t-1}$) has shape $(na, m)$ The input data at timestep t ...
1
vote
0answers
35 views

What is the computational complexity in terms of Big-O notation of a Gated Recurrent Unit Neural network?

I have been digging up of articles across the internet in context of computational complexity of GRU. Interestingly, I came across this article, http://cse.iitkgp.ac.in/~psraja/FNNs%20,RNNs%20,LSTM%...
0
votes
0answers
30 views

Why do we have a sigmoid function in the input layer in LSTMs? [closed]

I'm particularly confused about the sigmoid function in the forget and input layer. If we use a sigmoid in the forget layer to look at $h_{t-1}$ and $x_{t}$, and output a number between 0 and 1 for ...
0
votes
0answers
22 views

In LSTMs, how does the additive property enables better balancing of gradient values during backpropagation?

There are two sources that I'm using to to try and understand why LSTMs reduce the likelihood of the vanishing gradient problem associated with RNNs. Both of these sources mention the reason LSTMs are ...
1
vote
1answer
40 views

How does back-propagation through time work for optimizing the weights of a bi-directional RNN?

I am aware that back-propagation through time is used for training the recurrent neural network. But I am not able to understand how this happens for the bi-directional versions of the recurrent ...
2
votes
1answer
37 views

Any comparison between transformer and RNN+Attention on the same dataset?

I am wondering what is believed to be the reason for superiority of transformer? I see that some people believe because of the attention mechanism used, it’s able to capture much longer dependencies. ...
0
votes
0answers
17 views

Stateful many-to-many RNN generating artefacts at regular intervals

I am training a stateful LSTM network, with a time series consisting of about 500000 data points spread over 5 years. This time series is split up to batches of 100 timesteps, and fed into the network....
1
vote
2answers
43 views

Is it possible to classify the subject of a conversation?

I would like to classify the subject of a conversation. I could classify each messages of the conversation, but I will loose some imformation because of related messages. I also need to do it ...
1
vote
0answers
18 views

Literature on computational modelling involving neuronal ensemblies

Straying from the current trends in deep learning, there is an, arguably, interesting idea of neuronal ensembles possibly providing an alternative to the current "layered feature detectors" ...
0
votes
1answer
39 views

What are the rules behind vector product in gradient?

Let's suppose we have calculated the gradient and it came out to be $f(WX)(1-f(W X))X$, where $f()$ is the sigmoid function, $W$ of order $2\times2$ is the weight matrix, and $X$ is an input vector of ...
0
votes
0answers
39 views

How to input a given sequence to a transformer (or an RNN) with probability of occurrence?

I'm experimenting with music and transformers, and I have sequences $S$ of shape: $(B,L,N)$ where $B$ is the batch size, $L$ is the sequence length, and $N=12$ are the number of musical notes with ...
1
vote
0answers
13 views

What's the difference between RNNs and Feed Forward Neural Networks if a fixed size vector can preserve sequential information?

I was watching a Youtube video in which the problem of trying to predict the last word in a sentence was posed. The sentence was "I took my cat for a" and the last word was "walk"....
1
vote
1answer
48 views

What's a good neural network for this problem?

I am very new to the field of AI so please bear with me. Say there is a dice with three sides, -1,0 and 1, and I want to predict which side it lands on (so only one output is needed I guess). The ...
0
votes
1answer
41 views

What type of model should I fit to increase accuracy?

Currently, I'm working on 6-axis IMU(Inertial Measurment Unit) dataset. This dataset contain 6 axis IMU data of 7 different drivers. The Imu sensor attached on vehicle. The drivers drives same path. ...
1
vote
1answer
31 views

Choosing an AI method to recreate a given binary 2D image

If the title wan not very clear, I want a method to take an input image like this, [[0, 0, 0, 0], [1, 1, 1, 0], [1, 1, 1, 0], [0, 1, 1, 0]] and output the 2D ...
1
vote
0answers
20 views

Unix timestamps for Recurrent Neural Networks

I want to use RNN for classifying whole sequences of events, generated by website visitors. Each event has some categorical properties and a Unix timestamp: ...
1
vote
1answer
26 views

Can the normal equation be used to optimise the RNN's weights?

I have made an RNN from scratch in Tensorflow.js. In order to update my weights (without needing to calculate the derivatives), I thought of using the normal equation to find the optimal values for my ...
2
votes
0answers
65 views

Doing backpropagation in an Tensorflow.js Neural Network

I have a neural network (which I am making from scratch). In order to make the neural network "learn" I need to conduct back-propagation. Using the code at the below how would I conduct back-...
0
votes
0answers
16 views

Time Series Forecasting - Recurrent Neural Networks (tensorflow)

I am attempting to forecast a time series using tensorflow with the following code: ...
0
votes
0answers
12 views

Python code in LSTM to look at selective history

My dataset has 3 columns - a,b,c. Using b (and its history), I wish to predict c. Using list function and converting to array, I can tell python to look at last 20 b's for any b, as input to predict c....
0
votes
0answers
31 views

What are the most used and effective activation functions for sentiment classification with an recurrent neural network?

I am making an RNN for sentiment classification. What activation functions would you use in order to achieve this goal (excluding the one present in the output layer)?
2
votes
1answer
39 views

How do RNN's for sentiment classification deal with different sentence lengths?

I have been doing a course which teaches you about Deep Neural Networks, during one of the exercises I was made to make an RNN for sentiment classification which I did, but I did not understand how an ...
3
votes
1answer
103 views

How can Transformers handle arbitrary length input?

The transformer, introduced in the paper Attention Is All You Need, is a popular new neural network architecture that is commonly viewed as an alternative to recurrent neural networks, like LSTMs and ...
1
vote
0answers
12 views

Understanding graphs of the mean square error: relationships between val loss and train loss

I am currently working with some models aimed at predicting time series (89 days for training, 22 for testing), including a CNN LSTM and a convLSTM. When training these models, I had the following ...
0
votes
0answers
10 views

Finding Cycles in a State Sequence

Suppose I observe a set of states $\mathbf{X} = \{X_{1}, X_{2}, \ldots, X_{K}\}$ over time. I assume that there exist $M$ cycles $\mathbf{C} = \{C_{1}, C_{2}, \ldots, C_{M} \}$ in the observed state ...
0
votes
2answers
47 views

Is there a neural network that accepts both the current input and previous output?

I am quite new to neural networks. I am trying to implement in Python a neural network having only one hidden layer with $N$ neurons and $1$ output layer. The point is that I am analyzing time series ...
1
vote
0answers
19 views

Keras model accuracy not improving beyond threshold

I am currently working on a public project for the National Weather Model. We are experimenting with using a recurrent neural network to replace the output of a quadratic formula that is in use. The ...
0
votes
0answers
20 views

What are some solutions for dealing with time series data that are recorded at uneven intervals?

Let's say I have a time series data which is a bunch of observations that occur at different time stamps and intervals. For example, my observations come from a camera located at a traffic ...
0
votes
0answers
25 views

How to find the derivative of a dynamic neuron model, which depends on previous states of the neuron?

This is the equation where n denotes the current state, (n-1) denotes the state in the previous step etc. And to do back-propagation I need to find partial derivatives over each of the variables. For ...
3
votes
1answer
82 views

How to use text as an input for a neural network - regression problem? How many likes/claps an article will get

I am trying to predict the number of likes an article or a post will get using a NN. I have a dataframe with ~70,000 rows and 2 columns: "text" (predictor - strings of text) and "likes&...
1
vote
1answer
48 views

Recommendations or resources for neural network/deep learning for time series application?

I know there are quite a few good deep learning books out there, but most explain neural networks and deep learning via application on images. If there are examples/code, they are often done on the ...
3
votes
1answer
352 views

What exactly are the “parameters” in GPT-3's 175 billion parameters and how are they chosen/generated?

When I studied neural networks, parameters were learning rate, batch size etc. But even GPT3's ArXiv paper does not mention anything about what exactly the parameters are, but gives a small hint that ...
1
vote
0answers
37 views

How can one be sure that a particular neural network architecture would work?

Traditionally, when working with tabular data, one can be sure(or at least know) that a model works because the included features could explain a target variable, say "Price of a ticket" ...
0
votes
1answer
45 views

What is the big fuzz about SHA-RNN versus Transformers?

In his paper introducing SHA-RNN (https://arxiv.org/pdf/1911.11423.pdf) Stephen Merity states that neglecting one direction of research (in this case LSTMs) over another (transformers) merily because ...
0
votes
1answer
58 views

Input tensor shape order for RNN (PyTorch)

I am confused as to why the sequence length is the first dimension of the input tensor for an RNN when batch size is the first dimension for any other kind of network (Linear/CNN/etc.). This makes me ...
1
vote
2answers
73 views

Why the cost/loss starts to increase for some iterations during the training phase?

I am trying to build a recurrent neural network from scratch. It's a very simple model. I am trying to train it to predict two words (dogs and gods). While training, the value of cost function starts ...
0
votes
0answers
24 views

How to feed the LSTM with different length for the latest time step?

I am having a training data set for a time-series dataset like below: ...
2
votes
1answer
66 views

Why are RNNs used in some computer vision problems?

I am learning computer vision. When I was going through implementations of various computer vision projects, some OCR problems used GRU or LSTM, while some did not. I understand that RNNs are used ...
0
votes
0answers
41 views

What does the notation “for t=T to 1,−1 do” in terms of time steps, in deep recurrent q network?

In looking at an algorithm in the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning. Here is the full algorithm: What does the notation ...
1
vote
0answers
31 views

What is the role of embeddings in a deep recurrent Q network?

When describing the model architecture for a deep recurrent q network, the authors of the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning each agent consists of a recurrent ...
1
vote
1answer
43 views

Why does my “entropy generation” RNN do so badly?

I'm new to relatively RNNs, and I'm trying to train generative and guessing neural networks to produce sequences of real numbers that look random. My architecture looks like this (each "circle&...
2
votes
1answer
87 views

What is the difference between LSTM and fully connected LSTM?

I'm currently trying to understand the difference between a vanilla LSTM and a fully connected LSTM. In a paper I'm reading, the FC-LSTM gets introduced as FC-LSTM may be seen as a multivariate ...
1
vote
0answers
64 views

Do you have to add a dense layer onto the final layer of an LSTM?

If my understanding of an LSTM is correct then the output from each LSTM unit is the hidden state from that layer. For the final layer if I wanted to predict e.g. a scalar real number, would I want to ...
0
votes
0answers
35 views

Why in RL function approximators with recurrent structures can learn planning?

In the paper An Investigation of Model-Free Planning the authors use ConvLSTM to learn a planning function. In particular, for each input x_t at time-step ...
0
votes
0answers
24 views

Neural Network architecture for going from scalar input to time series outputs?

I have a problem where I know p features which are each scalar values and the output of 1 set of those features is a time history. Is there a specific neural network (NN) type architecture that can ...
2
votes
1answer
40 views

How do LSTM or GRU gates learn to specialize in their desired tasks?

While I was studying the equations for the computation inside GRU and LSTM units, I realized that although the different gates have different Weight matrices, their overall structure is the same. They ...

1
2 3 4 5