# Tag Info

15

Your question is quite broad, but here are some tips: For feedforward networks, see this question: @doug's answer has worked for me. There's one additional rule of thumb that helps for supervised learning problems. The upper bound on the number of hidden neurons that won't result in over-fitting is: $$N_h = \frac{N_s} {(\alpha * (N_i + N_o))}$$...

11

How does this method differ from the experience replay, as they both use past information in the training? What's the typical application of both techniques? Using a recurrent neural network is one way for an agent to build a model of hidden or unobserved state in order to improve its predictions when direct observations do not give enough information, but ...

8

From nvidia www (https://developer.nvidia.com/discover/lstm): Accelerating Long Short-Term Memory using GPUs The parallel processing capabilities of GPUs can accelerate the LSTM training and inference processes. GPUs are the de-facto standard for LSTM usage and deliver a 6x speedup during training and 140x higher throughput during inference when compared ...

7

I would recommend to start by reading this blogpost. You can probably cannibalise the code to create a RNN that takes in one statement of a dialogue and then proceeds to output the answer to that statement. That would be the easy version of your project, all without word vectors and thought vectors. You are just inputting characters, so typos don't need to ...

7

Select the number of hidden layers and number of memory cells in LSTM is always depend on application domain and context where you want to apply this LSTM. For hidden Layers. The introduction of hidden layer(s) makes it possible for the network to exhibit non-linear behaviour. The optimal number of hidden units could easily be smaller than the number of ...

7

An RNN or LSTM have the advantage of "remembering" the past inputs, to improve performance over prediction of a time-series data. If you use a neural network over like the past 500 characters, this may work but the network just treat the data as a bunch of data without any specific indication of time. The network can learn the time representation only ...

5

I think there are two parts to answering this question. First, about the specific paper that has been mentioned. The paper's title is hyperbolic, and probably written that way to get more people to read it. The paper itself does not make the claim that attention-based networks will supplant existing recurrent network architectures. Instead, it makes a more ...

5

Your scenario is common. The most straightforward approach is to subsample your data randomly. Unless your data or your model has strong bias, your performance to the smaller data set should be comparable. The accuracy might be lower, but the purpose is to do quick sanity check.

4

In general, there are no guidelines on how to determine the number of layers or the number of memory cells in a LSTM. The number of layers and cells required in an LSTM might depend on several aspects of the problem: The complexity of the dataset. The amount of features, number of data points etc. The the data generating process. Following example of how ...

4

I found that there are cuDNN accelerated cells in Keras for example: https://keras.io/layers/recurrent/#cudnnlstm They very fast. The normal LSTM cells are faster on CPU then on GPU. Also see here for a comparisem: https://wiki.eniak.de/ml/geschwindigkeitsvergleich_keras_lstm_und_cudnnlstm

3

After doing a bit of research I found that the LSTM whose gates perform convolutions is called ConvLSTM. The term CNN LSTM is loose and may mean stacking up LSTM on top of CNN for tasks like video classification Reddit thread discussing this

3

It was difficult to find because recurrent network designs predate LSTM extensions of that earlier idea by decades. Although the term recurrent was not yet used as a primary description of the technology advancement, recurrence was an essential feature of the theoretical treatment of artificial networks that learned actions in Attractor dynamics and ...

3

writing here my suggestion, because i haven't earned the right to comment yet. Your main "problem" could be your loss function. It converges, this is why your loss value is decreasing. So I suggest to let it maybe train longer. Alternatively you could change the loss function to fit your need. For example you could use: loss = tf.reduce_mean(tf.square(...

3

Take the sentence that was generated by your LSTM and feed it back into the LSTM as input. Then the LSTM will generate the next sentence. So the LSTM is using it's previous output as it's input. That's what makes it recursive. The intial word is just your base case. Also you should consider using GPT2 by open AI to do this. It's pretty impressive. https://...

2

This might work for your case but isn't necessarily true and depends on how much data the network goes through in an iteration. You should be able to test this by making a small change and training until 100 iterations and seeing if the performance significantly changes and if it can be predicted from the 20th iteration. Another way which may work for you ...

2

So the equation that you mentioned is used during the backward pass in which back proppogation is performed in order to make the neural network more accurate. I think you are talking about the state during the forward pass which is completely different. In the forward pass, the neural network is simply run in order to evaluate or it is simply used as a model....

2

So you want your network to represent those 3 values at each step as single composite value? I can't think of any better way than utilizing 3 LSTM units but attaching them to same write and read nodes of the enclosing network. In other words your assumption that it makes sense to keep all those 3 values together, gets hardcoded into your network by making 6 ...

2

Maybe you should have a look at this: https://wiki.inf.ed.ac.uk/twiki/pub/CSTR/ListenTerm1201415/sak2.pdf Here they show that 2 layers are nice, 5 layers are better and 7 layers are very hard to train.

2

Structure of LSTM Networks [Are] units/neurons in the hidden layers ... referred to as memory blocks and each memory block can contain multiple memory cells? Although an early guess at the way biological neurons work inspired the Perceptron design that later became multilayer (MLP) and then recurrent (RNN), these artificial network building blocks are ...

2

You compute the error in the same way as normal - treat the actual output as "ground truth" even if other options are possible. The LSTM will output probabilities of next character, it will not learn to associate a single "true" output except for rare circumstances (such as completing a word that can only be done in one way according to the training data). ...

2

In the perceptron design generally used in Artificial Neural Networks, we know precisely what a single neuron is capable of computing. It can compute a function $$f(x) = g(w^{\top} x),$$ where $x$ is a vector of inputs (may also be vector of activation levels in previous layer), $w$ is a vector of learned parameters, and $g$ is an activation function. We ...

2

In my opinion, there are many functions in our brain. Surely much more than the artificial neural network nowadays. I guess this is the field of brain science or cognitive psychology. Some brain structures may help for certain applications, but not all. Neural network though is a simplest form of our brain, but has the most general usages. On the other ...

2

Why would giving my AI more data make it perform worse? A lot of possible reasons: In forecasting, you could have a seasonality. If you have it exactly 3 times, then it is good. If you have it 3.5 times, it becomes worse because it overfits on the months which occur more often than the others. Data quality: In practice, improving the quality of data often ...

2

The cell of a perceptron was based on an oversimplified conception of a neuron. At the time, neural plasticity, timing factors in relation to activation, neurochemical pathways, and energy transit complexities in axons were unknown. The mapping of pulse transmission to basic algebra seemed unrealistic, so timing was ignored. Plasticity, timing, and regional ...

2

Is figure = code? No. Your figure shows a fully connected feed forward network (MLP). But in your code you are using a two layer LSTM with peepholes. For the visualization of LSTMs, blocks are usually used for each layer. Here is a figure of the LSTM with peepholes which is the base of the tensorflow implementation (Source: Paper, fig. 1). Why size 86? ...

2

Can the decoder in a transformer model be parallelized like the encoder? NO: Your understanding is completely right. In the decoder, the output of each step is fed to the bottom decoder in the next time step, just like an LSTM. Also, like in LSTMs, the self-attention layer needs to attend to earlier positions in the output sequence in order to compute ...

2

Congrats, you have invented 1d convolution. Convolution combined with RNN would have some advantage over just RNN. Think about the perception field. In this layer, you do aggregate $6$ values to one. Imagine two of them - it will be $36$ already, etc. But, in the end, you still need RNN at the end to aggregate a variable length to constant length.

2

It's more domain- or task-specific. There is no obvious baseline anymore because these models and this field has evolved into too large of an ecosystem. Nonetheless, I'll list a couple of notorious examples below. Image classification: MNIST CIFAR ImageNet Detection/segmentation: PascalVOC COCO CityScapes Pose estimation: MPII LEEDS ...

2

Assumptions Different model structures encode different assumptions - while we often make simplifying assumptions that aren't strictly correct, some assumptions are more wrong than others. For example, your proposed structure of "just pass the $X$ number of letters leading up to the last letter into an FFNN" makes an assumption that all the information ...

2

TL;DR Here is a beautiful explanation with diagrams: source To address: the cell state is essentially long term memory embedding (correct me if I'm wrong) The embedding can be long or short term and it is a vector. To answer: Why is the previous hidden state, current input and the bias put into a sigmoid function? Is there some special ...

Only top voted, non community-wiki answers of a minimum length are eligible