Questions tagged [recurrent-neural-networks]

For questions related to recurrent neural networks (RNNs), artificial neural networks that contain backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network. An RNN can be trained using back-propagation through time, such that these backward connections "memorize" previously seen inputs. Consequentially, RNNs are well suited to sequence prediction and similar tasks.

Filter by
Sorted by
Tagged with
2 votes
1 answer
56 views

How can I deal with random weights initialisation when predicting a time-series sine function?

I am training a simple RNN model in keras to predict a time series. The time series I am considering is just a sine function The task to solve is the following: ...
apt45's user avatar
  • 123
0 votes
0 answers
13 views

How can a RNN with 256 cells accept a input of any size?

I built a 3 layered RNN model with 256 cells each using torch. Input feature size is set to 40. Below give a basic Idea on the model. ...
D Star Let's Explore's user avatar
0 votes
0 answers
24 views

Seeking methods to incorporate arbitrary actuator faults for Control Optimization

I am working on a problem where a control method, backed by a Neural Network (NN), dictates the movement of a 1D actuator to influence a specific process. This actuator can move linearly within a set ...
IsolatedSushi's user avatar
1 vote
1 answer
23 views

LSTM with multiple data streams

I am working on the following problem: I have ~10 weather stations in somewhat approximate areas, at some points during the day (different for each station), I get readings of various data points (...
PenguinHook's user avatar
0 votes
0 answers
13 views

Feeding variable length of 2D image slices of the MRI into the deep neural network

I am trying to build a classifier that would predict the correct outcome (disease vs healthy) using a set of 2D slices derived from the 3D MRI scan. For each patient, based on the 3D scan, I am able ...
Oleg's user avatar
  • 101
0 votes
0 answers
29 views

Is my 1D signal using CNN & RNN regression reasonable?

I want to know if my impact-echo signals are proper with CNN or RNN regression model. I got some simulated signal, as following shows. In previous research, people mostly consider frequency or even ...
hui30319's user avatar
0 votes
0 answers
26 views

Does an RNN architecture exist where the output are actually samples drawn from a (non-parametric) probability distribution?

Does an RNN architecture exist where my input is a 10 dimensional real-valued vector and the output are 200 samples drawn from a probability distribution. In essence, the RNN is actually learning a ...
Akash Tiwari's user avatar
0 votes
0 answers
14 views

RNN - Time To Stop extimation given non-categorical events sequence

I have a dataset which contains sequences of event of the following type: Timestamp_s Timestamp_e Event_type I've preprocessed the data to create sequences of the shape [seq_size, unique_codes_count]...
GPU'njoyer's user avatar
0 votes
1 answer
42 views

Temporally Non-Aware RNN

I am trying to classify whether or not a specific object is in panoramic photos. The issue is, a panoramic photo can be any width, so the input to my neural network can't be fixed in that dimension. I'...
user avatar
0 votes
0 answers
11 views

Neural Network (NN) to predict the duration (in seconds) of a fault, how to recognize the elapsed time in prediction of an active fault?

I'm working on developing a neural network (NN) to predict the duration (in seconds) of a fault. However, I've encountered a couple of challenges: Model Performance: My neural network seems to be ...
Tools's user avatar
  • 1
0 votes
0 answers
19 views

How do I find a similar RNN as a starting point?

I am new to machine learning and neural networks and I want to create a neural network for a study project. I would like to create a RNN, that uses one (A) or several time series (with the same length,...
xlaub's user avatar
  • 1
0 votes
0 answers
47 views

What are the *non-cost-related* reasons RNN+Attention underperform Transformers?

There are obvious trainability and performance challenges with RNNs, such as having to process in serial and BPTT. But let's say we magically had an "optimal" set of weights for the RNN + ...
llllvvuu's user avatar
  • 146
1 vote
1 answer
40 views

How does the linear layer step work and what should I do at the end of the LSTM?

So basically I've read some text about LSTM, and luckily they mentioned the linear layer step at the end of the LSTM Process. However, they didn't explain how it works or what I would need to convert ...
Anish Kommireddy's user avatar
0 votes
0 answers
55 views

How to improve CRNN model(CTC loss) accuracy for OCR task?

I take this as baseline model. The main difference on RNN part, ...
4daJKong's user avatar
  • 101
0 votes
0 answers
30 views

LSTM Ensemble: Combine low, mid, and high frequency time series data

I am trying to implement time series classification, but I am struggling a bit with the fact that my multivariate data has mixed frequencies. I have about 10 variables that are updated every minute, ...
Ai4l2s's user avatar
  • 121
2 votes
0 answers
145 views

Why is it said transformers are more parallelizable than RNN's?

The parallelization of transformers and RNNs (Recurrent Neural Networks) is often discussed. It's commonly said that transformers are more parallelizable than RNNs. However, this is a rather vague ...
Erick Macias's user avatar
0 votes
1 answer
20 views

Handcraft RNN with attention to extract central element

I am trying to formulate an RNN that uses attention to easily detect the central element of a sequence. For an RNN alone this is not an easy task but with attention, it should be but I am not entirely ...
Daraan's user avatar
  • 101
0 votes
1 answer
620 views

Which preprocessing is the correct way to forecast time-series data using LSTM?

I just started to study time-series forecasting using RNN. I have a few months of time series data that was an hour unit. The data is a kind of percentage value of my little experiment and I would ...
orde.r's user avatar
  • 103
0 votes
0 answers
10 views

Neural translation model - alignment as a latent variable or as a trainable part of the model

I consider the neural machine translation method in https://arxiv.org/pdf/1409.0473.pdf. I understand that the intuition here, as the title says, is to jointly train alignment and translation. ...
kiriloff's user avatar
  • 121
0 votes
0 answers
22 views

How to use deep relative trust to measure the distance between two RNNs, and between two transformers?

I want to measure how different two networks, i.e., between two RNNs and between two transformers. I read that Deep relative trust can be used to measure the distance between two NNs. Can it be used ...
Sanyou's user avatar
  • 165
0 votes
1 answer
35 views

RNNs - Is the recurrence at the layer-level or at the network-level?

I am confused by where the recurrence happens in RNNs, especially in the context of deep neural networks. I am trying to transform an ordinary neural network into a recurrent one from scratch. ...
BPDev's user avatar
  • 103
0 votes
0 answers
34 views

Is this lstm diagram correct?

I made an LSTM diagram, but do not know if it is correct. Can you point out any errors in case there are any?
David H. J.'s user avatar
3 votes
1 answer
48 views

In a Recurrent Neural Network, what are the inputs to a node in a mutli-layer RNN?

I'm trying to work through a project where I'm writing my own RNN in C++ - not using any libraries. Basically I have an Input layer - 2 hidden layers - and then an output layer. In a given layer, each ...
Mike Arsenault's user avatar
0 votes
3 answers
100 views

Can an RNN predict a sinus curve with no input?

I read a number of tutorials on how to make an RNN (simple, LSTM, etc.) that predicts a sinus curve. They all use as an input (x) in every step a set of past sinus values. I am wondering if ...
jollytall's user avatar
1 vote
0 answers
88 views

In terms of explainability, is attentive RNN easier to explain than the transformer?

Although the multi-headed attention block of the transformer allows the model to be more expressive (and therefore perform better), it is remarkably more difficult to decompose and therefore to ...
hH1sG0n3's user avatar
  • 191
13 votes
4 answers
5k views

Why LLMs and RNNs learn so fast during inference but, ironically, are so slow during training?

Why LLMs learn so fast during inference, but, ironically, are so slow during training? That is, if you teach an AI a new concept in a prompt, it will learn and use the concept perfectly and flawless, ...
MaiaVictor's user avatar
4 votes
2 answers
847 views

LLM-like architecture capable of dynamically learning from its own output

Language Learning Models (LLMs) have demonstrated remarkable capabilities in quick learning during inference. They can effectively grasp a concept from a single example and generate relevant outputs. ...
MaiaVictor's user avatar
1 vote
1 answer
69 views

Recognize patterns within random sequences

I am familiar with ANNs as I studied them back in the days for regression and currently I'm working with CNN's for image recognition. But recently I was reading more about pattern recognition in ...
FELIPE_RIBAS's user avatar
0 votes
1 answer
585 views

Difference between sequence length and hidden size in LSTM

It does not come clear to me how the seq_length is not the exact same as the hidden_size in LSTMs. For example, in the next ...
moth123's user avatar
  • 31
0 votes
1 answer
63 views

Seq2seq with RNNs, how is the training loop performed?

How do we train a seq2seq rnn training? We input a sentence that needs to be translated. We encode it sequentially. Then the first decoder outputs the first word with probabilities. We do a gradient ...
FluidMechanics Potential Flows's user avatar
0 votes
0 answers
7 views

How to create/train a binary classification model for checking candidate phrases?

Let's say I have sentences like "He called me a silly sausage when I made a stupid joke", and I want to identify/extract all swear words and the like (here: "silly sausage"). I ...
Christian's user avatar
  • 101
0 votes
0 answers
64 views

How can a neural network learn to predict shapes or sets using only sampled points?

For $t=0,1,\dots$, consider a parameter $x_t \in ${$1, \dots, n$}, where $n \in \mathbb{N}$, and a shape $S(x_t)$ on an $m \times m$ square grid $G$. Let's denote the status of a cell $(i,j) \in G$ at ...
adidave's user avatar
1 vote
1 answer
35 views

Many To One LSTM - Can I Use the Same Sequence as Input from Previous Timesteps?

I'm new to LSTMs, and I'm trying to do a basic timeseries prediction using stock prices. However, I'm a bit confused as to how the LSTM is supposed to remember outputs from previous timesteps when it ...
Krusty the Clown's user avatar
0 votes
1 answer
38 views

How to improve classification accuracy in TF deep neural network model?

I need help in increasing the accuracy of a classification model using Neural Networks on Tensorflow. I am trying to train a model on sequential data ...
Fr_nkenstien's user avatar
2 votes
1 answer
127 views

Why do training and fixing a reservoir yield very similar results (in an echo state network)?

Disclaimer: I asked this question 2 days ago in Cross Validated, but it has been left unanswered. I am trying to better understand how echo state networks work. To see, how fixing the weights of the ...
user avatar
1 vote
0 answers
50 views

About RNN followed by dense layer

In RNN we do get one output for each time stamp of input right i.e. if we give input as "I am Good" we get three outputs representing I followed by am and Good so if we connect a dense layer ...
Sowrabh meduri's user avatar
0 votes
0 answers
86 views

When should I think of using the Forward Forward algorithm?

Recently I read the paper named "The Forward-Forward Algorithm: Some Preliminary Investigations" and I was wondering what cases I should think of using it on. So according to the paper If we ...
HAMDI ABDERRAHMENE's user avatar
0 votes
0 answers
9 views

Constrain or not constrain overlapping architectural choices when wanting to compare deep neural architectures

I want to compare various deep (recurrent) neural architectures and was wondering what the best approach is. The models in question all use several LSTMs/GRUs etc. Fine-tune all models fully and ...
Robin van Hoorn's user avatar
0 votes
0 answers
56 views

What AI model is best for finding optimal sequence taking into account sequence length?

I'm trying to find the best AI model to solve the following problem: I have a list of abilities: Ability 1 - Deals X1 amount of damage. Can be used every Y1 turns. Ability 2 - Deals X2 amount of ...
Gabrielius's user avatar
1 vote
1 answer
798 views

Would it be possible to involve a proof assistant in the process of training a LLM?

LLMs like GPT-3 have been shown capable of outputting highly complex code. Sadly, actually using them to replace a programmer's job has two major caveats: LLMs are notoriously bad at producing ...
MaiaVictor's user avatar
0 votes
0 answers
36 views

How to evaluate a neural network that has recurrent connections

I was attempting to implement NEAT but I am facing a slight problem. how can I get the order for which to calculate the output of each neuron with recurrent connections present? I thought if a method ...
Mahmoud Hany's user avatar
1 vote
1 answer
36 views

How to deal with varying number of input images?

Im trying to use Deep-Learning to recognize breast cancer on Mammography Images. But in the dataset every patient has a different (1-4) number of images taken. How can i deal with that? Generally i ...
Patrick G Patrick's user avatar
1 vote
1 answer
50 views

Which calculation to use for GRU

Im doing trying to implement GRU in my own Neural Network Library but when I did some research i stumbled on some inconsistencies. When calculating a cell there are as many legitimate resources which ...
Johannes K.'s user avatar
0 votes
1 answer
55 views

What underlying network is typically meant with ResNET?

When people talk about a ResNet architecture, they are talking about a neural network architecture with skip connections. But what basis network are they typically referring to? Feedforward-networks ...
postnubilaphoebus's user avatar
0 votes
0 answers
32 views

Modeling the previous inputs to affect next output in Machine learning

I am working on a dataset contains one output variable and a number of input variables.The data looks like the following: Y, X1, X2, X3, X4 7, 5, 0.7, 8, 9 3, 6, 0.3, 9, 9 .... Where Y is the output ...
Yazan Alatoom's user avatar
0 votes
1 answer
55 views

How does not learning far inputs make the RNN forget far inputs?

I am totally aware of the problem of the vanishing gradient. It usually occurs with vanilla RNN, where with a long sequence of data, the gradient will vanish or explode for far input sequence, and ...
John adams's user avatar
3 votes
1 answer
531 views

Initial State of RNN

Can I initialize the initial state of my RNN to be non-zero? I have some initial condition of the sequence and I want to use this initial condition as the initial state.
wrek's user avatar
  • 183
0 votes
1 answer
141 views

Extremely good results in RNN-LSTM python code!! How can this happen?

I am using this code here: https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/ and more specifically the python code under the (1st) paragraph "...
just_learning's user avatar
1 vote
1 answer
268 views

Batching together similar length sequences to avoid padding and packing

I am training an RNN in PyTorch to produce captions for images. It's a pretty standard architecture – the image is processed by a pre-trained InceptionV3 to extract features, the recurrent module ...
czypsu's user avatar
  • 111
1 vote
0 answers
47 views

Surrogate model to produce time series from parameter set

Say I have a model $M$ that takes in a parameter vector $\beta$, and produces a (numerical) time series. This could be a complicated model (e.g. a bespoke enzyme reaction model), or something simple ...
Mich55's user avatar
  • 11

1
2 3 4 5
8