Questions tagged [recurrent-neural-networks]
For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.
354
questions
2
votes
1
answer
56
views
How can I deal with random weights initialisation when predicting a time-series sine function?
I am training a simple RNN model in keras to predict a time series.
The time series I am considering is just a sine function
The task to solve is the following:
...
- 123
0
votes
0
answers
13
views
How can a RNN with 256 cells accept a input of any size?
I built a 3 layered RNN model with 256 cells each using torch. Input feature size is set to 40. Below give a basic Idea on the model.
...
0
votes
0
answers
24
views
Seeking methods to incorporate arbitrary actuator faults for Control Optimization
I am working on a problem where a control method, backed by a Neural Network (NN), dictates the movement of a 1D actuator to influence a specific process. This actuator can move linearly within a set ...
1
vote
1
answer
23
views
LSTM with multiple data streams
I am working on the following problem:
I have ~10 weather stations in somewhat approximate areas, at some points during the day (different for each station), I get readings of various data points (...
- 11
0
votes
0
answers
13
views
Feeding variable length of 2D image slices of the MRI into the deep neural network
I am trying to build a classifier that would predict the correct outcome (disease vs healthy) using a set of 2D slices derived from the 3D MRI scan. For each patient, based on the 3D scan, I am able ...
- 101
0
votes
0
answers
29
views
Is my 1D signal using CNN & RNN regression reasonable?
I want to know if my impact-echo signals are proper with CNN or RNN regression model.
I got some simulated signal, as following shows.
In previous research, people mostly consider frequency or even ...
- 1
0
votes
0
answers
26
views
Does an RNN architecture exist where the output are actually samples drawn from a (non-parametric) probability distribution?
Does an RNN architecture exist where my input is a 10 dimensional real-valued vector and the output are 200 samples drawn from a probability distribution.
In essence, the RNN is actually learning a ...
0
votes
0
answers
14
views
RNN - Time To Stop extimation given non-categorical events sequence
I have a dataset which contains sequences of event of the following type:
Timestamp_s
Timestamp_e
Event_type
I've preprocessed the data to create sequences of the shape [seq_size, unique_codes_count]...
0
votes
1
answer
42
views
Temporally Non-Aware RNN
I am trying to classify whether or not a specific object is in panoramic photos. The issue is, a panoramic photo can be any width, so the input to my neural network can't be fixed in that dimension.
I'...
user55176
0
votes
0
answers
11
views
Neural Network (NN) to predict the duration (in seconds) of a fault, how to recognize the elapsed time in prediction of an active fault?
I'm working on developing a neural network (NN) to predict the duration (in seconds) of a fault. However, I've encountered a couple of challenges:
Model Performance: My neural network seems to be ...
- 1
0
votes
0
answers
19
views
How do I find a similar RNN as a starting point?
I am new to machine learning and neural networks and I want to create a neural network for a study project.
I would like to create a RNN, that uses one (A) or several time series (with the same length,...
- 1
0
votes
0
answers
47
views
What are the *non-cost-related* reasons RNN+Attention underperform Transformers?
There are obvious trainability and performance challenges with RNNs, such as having to process in serial and BPTT. But let's say we magically had an "optimal" set of weights for the RNN + ...
- 146
1
vote
1
answer
40
views
How does the linear layer step work and what should I do at the end of the LSTM?
So basically I've read some text about LSTM, and luckily they mentioned the linear layer step at the end of the LSTM Process. However, they didn't explain how it works or what I would need to convert ...
- 111
0
votes
0
answers
54
views
How to improve CRNN model(CTC loss) accuracy for OCR task?
I take this as baseline model. The main difference on RNN part,
...
- 101
0
votes
0
answers
30
views
LSTM Ensemble: Combine low, mid, and high frequency time series data
I am trying to implement time series classification, but I am struggling a bit with the fact that my multivariate data has mixed frequencies. I have about 10 variables that are updated every minute, ...
- 121
2
votes
0
answers
145
views
Why is it said transformers are more parallelizable than RNN's?
The parallelization of transformers and RNNs (Recurrent Neural Networks) is often discussed. It's commonly said that transformers are more parallelizable than RNNs. However, this is a rather vague ...
- 21
0
votes
1
answer
20
views
Handcraft RNN with attention to extract central element
I am trying to formulate an RNN that uses attention to easily detect the central element of a sequence. For an RNN alone this is not an easy task but with attention, it should be but I am not entirely ...
- 101
0
votes
1
answer
615
views
Which preprocessing is the correct way to forecast time-series data using LSTM?
I just started to study time-series forecasting using RNN.
I have a few months of time series data that was an hour unit.
The data is a kind of percentage value of my little experiment and I would ...
- 103
0
votes
0
answers
10
views
Neural translation model - alignment as a latent variable or as a trainable part of the model
I consider the neural machine translation method in https://arxiv.org/pdf/1409.0473.pdf.
I understand that the intuition here, as the title says, is to jointly train alignment and translation. ...
- 121
0
votes
0
answers
22
views
How to use deep relative trust to measure the distance between two RNNs, and between two transformers?
I want to measure how different two networks, i.e., between two RNNs and between
two transformers. I read that Deep relative trust can be used to measure the distance between two NNs. Can it be used ...
- 165
0
votes
1
answer
35
views
RNNs - Is the recurrence at the layer-level or at the network-level?
I am confused by where the recurrence happens in RNNs, especially in the context of deep neural networks. I am trying to transform an ordinary neural network into a recurrent one from scratch.
...
- 103
0
votes
0
answers
34
views
Is this lstm diagram correct?
I made an LSTM diagram, but do not know if it is correct. Can you point out any errors in case there are any?
- 101
3
votes
1
answer
48
views
In a Recurrent Neural Network, what are the inputs to a node in a mutli-layer RNN?
I'm trying to work through a project where I'm writing my own RNN in C++ - not using any libraries. Basically I have an Input layer - 2 hidden layers - and then an output layer. In a given layer, each ...
0
votes
3
answers
100
views
Can an RNN predict a sinus curve with no input?
I read a number of tutorials on how to make an RNN (simple, LSTM, etc.) that predicts a sinus curve. They all use as an input (x) in every step a set of past sinus values.
I am wondering if ...
- 11
1
vote
0
answers
88
views
In terms of explainability, is attentive RNN easier to explain than the transformer?
Although the multi-headed attention block of the transformer allows the model to be more expressive (and therefore perform better), it is remarkably more difficult to decompose and therefore to ...
- 191
12
votes
4
answers
5k
views
Why LLMs and RNNs learn so fast during inference but, ironically, are so slow during training?
Why LLMs learn so fast during inference, but, ironically, are so slow during training? That is, if you teach an AI a new concept in a prompt, it will learn and use the concept perfectly and flawless, ...
- 355
4
votes
2
answers
845
views
LLM-like architecture capable of dynamically learning from its own output
Language Learning Models (LLMs) have demonstrated remarkable capabilities in quick learning during inference. They can effectively grasp a concept from a single example and generate relevant outputs. ...
- 355
1
vote
1
answer
69
views
Recognize patterns within random sequences
I am familiar with ANNs as I studied them back in the days for regression and currently I'm working with CNN's for image recognition. But recently I was reading more about pattern recognition in ...
- 111
0
votes
1
answer
581
views
Difference between sequence length and hidden size in LSTM
It does not come clear to me how the seq_length is not the exact same as the hidden_size in LSTMs.
For example, in the next ...
- 31
0
votes
1
answer
63
views
Seq2seq with RNNs, how is the training loop performed?
How do we train a seq2seq rnn training?
We input a sentence that needs to be translated. We encode it sequentially. Then the first decoder outputs the first word with probabilities. We do a gradient ...
0
votes
0
answers
7
views
How to create/train a binary classification model for checking candidate phrases?
Let's say I have sentences like "He called me a silly sausage when I made a stupid joke", and I want to identify/extract all swear words and the like (here: "silly sausage").
I ...
- 101
0
votes
0
answers
64
views
How can a neural network learn to predict shapes or sets using only sampled points?
For $t=0,1,\dots$, consider a parameter $x_t \in ${$1, \dots, n$}, where $n \in \mathbb{N}$, and a shape $S(x_t)$ on an $m \times m$ square grid $G$. Let's denote the status of a cell $(i,j) \in G$ at ...
- 1
1
vote
1
answer
35
views
Many To One LSTM - Can I Use the Same Sequence as Input from Previous Timesteps?
I'm new to LSTMs, and I'm trying to do a basic timeseries prediction using stock prices. However, I'm a bit confused as to how the LSTM is supposed to remember outputs from previous timesteps when it ...
- 111
0
votes
1
answer
38
views
How to improve classification accuracy in TF deep neural network model?
I need help in increasing the accuracy of a classification model using Neural Networks on Tensorflow.
I am trying to train a model on sequential data ...
- 457
2
votes
1
answer
127
views
Why do training and fixing a reservoir yield very similar results (in an echo state network)?
Disclaimer: I asked this question 2 days ago in Cross Validated, but it has been left unanswered.
I am trying to better understand how echo state networks work. To see, how fixing the weights of the ...
user68128
1
vote
0
answers
50
views
About RNN followed by dense layer
In RNN we do get one output for each time stamp of input right i.e. if we give input as "I am Good" we get three outputs representing I followed by am and Good so if we connect a dense layer ...
0
votes
0
answers
86
views
When should I think of using the Forward Forward algorithm?
Recently I read the paper named "The Forward-Forward Algorithm: Some Preliminary Investigations" and I was wondering what cases I should think of using it on.
So according to the paper If we ...
0
votes
0
answers
9
views
Constrain or not constrain overlapping architectural choices when wanting to compare deep neural architectures
I want to compare various deep (recurrent) neural architectures and was wondering what the best approach is. The models in question all use several LSTMs/GRUs etc.
Fine-tune all models fully and ...
- 2,103
0
votes
0
answers
56
views
What AI model is best for finding optimal sequence taking into account sequence length?
I'm trying to find the best AI model to solve the following problem:
I have a list of abilities:
Ability 1 - Deals X1 amount of damage. Can be used every Y1 turns.
Ability 2 - Deals X2 amount of ...
1
vote
1
answer
797
views
Would it be possible to involve a proof assistant in the process of training a LLM?
LLMs like GPT-3 have been shown capable of outputting highly complex code. Sadly, actually using them to replace a programmer's job has two major caveats:
LLMs are notoriously bad at producing ...
- 355
0
votes
0
answers
36
views
How to evaluate a neural network that has recurrent connections
I was attempting to implement NEAT but I am facing a slight problem.
how can I get the order for which to calculate the output of each neuron with recurrent connections present?
I thought if a method ...
1
vote
1
answer
36
views
How to deal with varying number of input images?
Im trying to use Deep-Learning to recognize breast cancer on Mammography Images. But in the dataset every patient has a different (1-4) number of images taken. How can i deal with that? Generally i ...
1
vote
1
answer
50
views
Which calculation to use for GRU
Im doing trying to implement GRU in my own Neural Network Library but when I did some research i stumbled on some inconsistencies.
When calculating a cell there are as many legitimate resources which ...
- 38
0
votes
1
answer
55
views
What underlying network is typically meant with ResNET?
When people talk about a ResNet architecture, they are talking about a neural network architecture with skip connections. But what basis network are they typically referring to? Feedforward-networks ...
0
votes
0
answers
30
views
Modeling the previous inputs to affect next output in Machine learning
I am working on a dataset contains one output variable and a number of input variables.The data looks like the following:
Y, X1, X2, X3, X4
7, 5, 0.7, 8, 9
3, 6, 0.3, 9, 9
....
Where Y is the output ...
0
votes
1
answer
55
views
How does not learning far inputs make the RNN forget far inputs?
I am totally aware of the problem of the vanishing gradient. It usually occurs with vanilla RNN, where with a long sequence of data, the gradient will vanish or explode for far input sequence, and ...
- 103
3
votes
1
answer
529
views
Initial State of RNN
Can I initialize the initial state of my RNN to be non-zero? I have some initial condition of the sequence and I want to use this initial condition as the initial state.
- 183
0
votes
1
answer
141
views
Extremely good results in RNN-LSTM python code!! How can this happen?
I am using this code here: https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/ and more specifically the python code under the (1st) paragraph "...
- 105
1
vote
1
answer
267
views
Batching together similar length sequences to avoid padding and packing
I am training an RNN in PyTorch to produce captions for images. It's a pretty standard architecture – the image is processed by a pre-trained InceptionV3 to extract features, the recurrent module ...
- 111
1
vote
0
answers
47
views
Surrogate model to produce time series from parameter set
Say I have a model $M$ that takes in a parameter vector $\beta$, and produces a (numerical) time series. This could be a complicated model (e.g. a bespoke enzyme reaction model), or something simple ...
- 11