20

Your question is quite broad, but here are some tips. Specifically for LSTMs, see this Reddit discussion Does the number of layers in an LSTM network affect its ability to remember long patterns? The main point is that there is usually no rule for the number of hidden nodes you should use, it is something you have to figure out for each case by trial and ...


14

How does this method differ from the experience replay, as they both use past information in the training? What's the typical application of both techniques? Using a recurrent neural network is one way for an agent to build a model of hidden or unobserved state in order to improve its predictions when direct observations do not give enough information, but ...


8

The selection of the number of hidden layers and the number of memory cells in LSTM probably depends on the application domain and context where you want to apply this LSTM. The optimal number of hidden units could be smaller than the number of inputs. AFAIK, there is no rule like multiply the number of inputs with $N$. If you have a lot of training ...


8

From nvidia www (https://developer.nvidia.com/discover/lstm): Accelerating Long Short-Term Memory using GPUs The parallel processing capabilities of GPUs can accelerate the LSTM training and inference processes. GPUs are the de-facto standard for LSTM usage and deliver a 6x speedup during training and 140x higher throughput during inference when compared ...


8

I'll list some bullet points of the main innovations introduced by transformers , followed by bullet points of the main characteristics of the other architectures you mentioned, so we can then compared them. Transformers Transformes (Attention is all you need) were introduced in the context of machine translation with the purpose to avoid recursion in order ...


7

I would recommend to start by reading this blogpost. You can probably cannibalise the code to create a RNN that takes in one statement of a dialogue and then proceeds to output the answer to that statement. That would be the easy version of your project, all without word vectors and thought vectors. You are just inputting characters, so typos don't need to ...


7

An RNN or LSTM have the advantage of "remembering" the past inputs, to improve performance over prediction of a time-series data. If you use a neural network over like the past 500 characters, this may work but the network just treat the data as a bunch of data without any specific indication of time. The network can learn the time representation only ...


6

I think there are two parts to answering this question. First, about the specific paper that has been mentioned. The paper's title is hyperbolic, and probably written that way to get more people to read it. The paper itself does not make the claim that attention-based networks will supplant existing recurrent network architectures. Instead, it makes a more ...


6

In general, there are no guidelines on how to determine the number of layers or the number of memory cells in an LSTM. The number of layers and cells required in an LSTM might depend on several aspects of the problem: The complexity of the dataset, such as the number of features, the number of data points, etc. The data-generating process. For example, ...


5

Your scenario is common. The most straightforward approach is to subsample your data randomly. Unless your data or your model has strong bias, your performance to the smaller data set should be comparable. The accuracy might be lower, but the purpose is to do quick sanity check.


4

I found that there are cuDNN accelerated cells in Keras for example: https://keras.io/layers/recurrent/#cudnnlstm They very fast. The normal LSTM cells are faster on CPU then on GPU. Also see here for a comparisem: https://wiki.eniak.de/ml/geschwindigkeitsvergleich_keras_lstm_und_cudnnlstm


4

After doing a bit of research I found that the LSTM whose gates perform convolutions is called ConvLSTM. The term CNN LSTM is loose and may mean stacking up LSTM on top of CNN for tasks like video classification Reddit thread discussing this


4

Have a look at the paper Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling (2014), where different LSTM architectures are compared. In the abstract, the authors write the following. We show that a two-layer deep LSTM RNN where each LSTM layer has a linear recurrent projection layer can exceed state-of-the-art ...


4

The diagram you show works at least partially for describing both individual neurons and layers of those neurons. However, the "incoming" data lines on the left represent all inputs under consideration, typically a vector of all inputs to the cell. That includes all data from current time steps (from input layer or earlier LSTM or time-distributed layers) - ...


4

I would say that the logic behind the introduction was more empirical than technical. The only difference between LSTM and Bi-LSTM is the possibility for Bi-LSTM to leverage future context chunks to learn better representations of single words. There is no special training step or units added, the idea is just to read a sentence forward and backward to ...


3

So the equation that you mentioned is used during the backward pass in which back proppogation is performed in order to make the neural network more accurate. I think you are talking about the state during the forward pass which is completely different. In the forward pass, the neural network is simply run in order to evaluate or it is simply used as a model....


3

It was difficult to find because recurrent network designs predate LSTM extensions of that earlier idea by decades. Although the term recurrent was not yet used as a primary description of the technology advancement, recurrence was an essential feature of the theoretical treatment of artificial networks that learned actions in Attractor dynamics and ...


3

writing here my suggestion, because i haven't earned the right to comment yet. Your main "problem" could be your loss function. It converges, this is why your loss value is decreasing. So I suggest to let it maybe train longer. Alternatively you could change the loss function to fit your need. For example you could use: loss = tf.reduce_mean(tf.square(...


3

Take the sentence that was generated by your LSTM and feed it back into the LSTM as input. Then the LSTM will generate the next sentence. So the LSTM is using it's previous output as it's input. That's what makes it recursive. The intial word is just your base case. Also you should consider using GPT2 by open AI to do this. It's pretty impressive. https://...


3

The sentences coming from the same document, author, etc., are unlikely to be independent, that is, the occurrence of a sentence $s_i$ in a certain document $d$ is likely correlated with the occurrence of another sentence $s_j$. If they are not independent, they can also not be independent and identically distributed (which is a stronger condition). The same ...


3

GPUs are able to execute a huge amount of similar and simple instructions (floating point operations like addition and multiplication) in parallel. In contrast to a CPU which is able to execute a few complex tasks sequentially very quick. Therefore GPUs are very good at doing vector & matrix operations. If you look at the operations performed inside a ...


2

This might work for your case but isn't necessarily true and depends on how much data the network goes through in an iteration. You should be able to test this by making a small change and training until 100 iterations and seeing if the performance significantly changes and if it can be predicted from the 20th iteration. Another way which may work for you ...


2

So you want your network to represent those 3 values at each step as single composite value? I can't think of any better way than utilizing 3 LSTM units but attaching them to same write and read nodes of the enclosing network. In other words your assumption that it makes sense to keep all those 3 values together, gets hardcoded into your network by making 6 ...


2

These newer RNNs (LSTMs and GRUs) have greater memory control, allowing previous values to persist or to be reset as necessary for many sequences of steps, avoiding "gradient decay" or eventual degradation of the values passed from step to step. LSTM and GRU networks make this memory control possible with memory blocks and structures called "gates" that pass ...


2

Structure of LSTM Networks [Are] units/neurons in the hidden layers ... referred to as memory blocks and each memory block can contain multiple memory cells? Although an early guess at the way biological neurons work inspired the Perceptron design that later became multilayer (MLP) and then recurrent (RNN), these artificial network building blocks are not ...


2

You compute the error in the same way as normal - treat the actual output as "ground truth" even if other options are possible. The LSTM will output probabilities of next character, it will not learn to associate a single "true" output except for rare circumstances (such as completing a word that can only be done in one way according to the training data). ...


2

In the perceptron design generally used in Artificial Neural Networks, we know precisely what a single neuron is capable of computing. It can compute a function $$f(x) = g(w^{\top} x),$$ where $x$ is a vector of inputs (may also be vector of activation levels in previous layer), $w$ is a vector of learned parameters, and $g$ is an activation function. We ...


2

Why would giving my AI more data make it perform worse? A lot of possible reasons: In forecasting, you could have a seasonality. If you have it exactly 3 times, then it is good. If you have it 3.5 times, it becomes worse because it overfits on the months which occur more often than the others. Data quality: In practice, improving the quality of data often ...


2

The cell of a perceptron was based on an oversimplified conception of a neuron. At the time, neural plasticity, timing factors in relation to activation, neurochemical pathways, and energy transit complexities in axons were unknown. The mapping of pulse transmission to basic algebra seemed unrealistic, so timing was ignored. Plasticity, timing, and regional ...


2

Is figure = code? No. Your figure shows a fully connected feed forward network (MLP). But in your code you are using a two layer LSTM with peepholes. For the visualization of LSTMs, blocks are usually used for each layer. Here is a figure of the LSTM with peepholes which is the base of the tensorflow implementation (Source: Paper, fig. 1). Why size 86? ...


Only top voted, non community-wiki answers of a minimum length are eligible