20

Your question is quite broad, but here are some tips. Specifically for LSTMs, see this Reddit discussion Does the number of layers in an LSTM network affect its ability to remember long patterns? The main point is that there is usually no rule for the number of hidden nodes you should use, it is something you have to figure out for each case by trial and ...


9

A recurrent neural network (RNN) is an artificial neural network that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN). The adjective "recurrent" thus refers to this backward or self-connections, which create loops in these networks. An RNN can be trained using back-...


8

The selection of the number of hidden layers and the number of memory cells in LSTM probably depends on the application domain and context where you want to apply this LSTM. The optimal number of hidden units could be smaller than the number of inputs. AFAIK, there is no rule like multiply the number of inputs with $N$. If you have a lot of training ...


8

I'll list some bullet points of the main innovations introduced by transformers , followed by bullet points of the main characteristics of the other architectures you mentioned, so we can then compared them. Transformers Transformes (Attention is all you need) were introduced in the context of machine translation with the purpose to avoid recursion in order ...


7

I would recommend to start by reading this blogpost. You can probably cannibalise the code to create a RNN that takes in one statement of a dialogue and then proceeds to output the answer to that statement. That would be the easy version of your project, all without word vectors and thought vectors. You are just inputting characters, so typos don't need to ...


7

There is a technique called Pruning in neural networks, which is used just for this same purpose. The pruning is done on the number of hidden layers. The process is very similar to the pruning process of decision trees. The pruning process is done as follows: Train a large, densely connected, network with a standard training algorithm Examine the trained ...


7

Basically, a CNN saves a set of weights and applies them spatially. For example, in a layer, I could have 32 sets of weights (also called feature maps). Each set of weights is a 3x3 block, meaning I have 3x3x32=288 weights for that layer. If you gave me an input image, for each 3x3 map, I slide it across all the pixels in the image, multiplying the regions ...


7

An RNN or LSTM have the advantage of "remembering" the past inputs, to improve performance over prediction of a time-series data. If you use a neural network over like the past 500 characters, this may work but the network just treat the data as a bunch of data without any specific indication of time. The network can learn the time representation only ...


6

Recurrent neural networks (RNNs) are artificial neural networks (ANNs) that have one or more recurrent (or cyclic) connections, as opposed to just having feed-forward connections, like a feed-forward neural network (FFNN). These cyclic connections are used to keep track of temporal relations or dependencies between the elements of a sequence. Hence, RNNs ...


6

In general, there are no guidelines on how to determine the number of layers or the number of memory cells in an LSTM. The number of layers and cells required in an LSTM might depend on several aspects of the problem: The complexity of the dataset, such as the number of features, the number of data points, etc. The data-generating process. For example, ...


6

I think there are two parts to answering this question. First, about the specific paper that has been mentioned. The paper's title is hyperbolic, and probably written that way to get more people to read it. The paper itself does not make the claim that attention-based networks will supplant existing recurrent network architectures. Instead, it makes a more ...


6

Recurrent neural networks (RNNs) are a class of artificial neural network architecture inspired by the cyclical connectivity of neurons in the brain. It uses iterative function loops to store information. Difference with traditional Neural networks using pictures from this book: And, an RNN: Notice the difference -- feedforward neural networks' ...


5

The approach you describe is called neuromorphic computing and it's quite a busy field. IBM's TrueNorth even has spiking neurons. The main problem with these projects is that nobody quite knows what to do with them yet. These projects don't try to create chips that are optimised to run a neural network. That would certainly be possible, but the ...


5

RNN's stand for Recurrent Neural Networks which is, in fact, Deep Learning. There has to be a loss since you're dealing with supervised learning and the typical loss metrics used are the same as you would see in feedforward networks (usually binary cross-entropy), the main difference being loss would be calculated between the true label at a particular time ...


4

You are right. I think you are just misinterpreting the part of the sentence ('specifically LSTMs'). LSTMs are an example of a popular type of RNN. RNNs and CNNs are different architectures but they can be used together. Here is another sentence with the same structure: It is clear than dogs, specifically corgis, and cats are very common in online memes.


4

Have a look at the paper Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling (2014), where different LSTM architectures are compared. In the abstract, the authors write the following. We show that a two-layer deep LSTM RNN where each LSTM layer has a linear recurrent projection layer can exceed state-of-the-art ...


4

If neurons and synapses can be implemented using transistors, I hope you are not talking about the neural networks which are currently winning all competitions in machine learning (MLPs, CNNs, RNNs, Deep Residual Networks, ...). Those were once used as a model for neurons, but they are only very loosely related to what happens in real brain cells. Spiking ...


4

I assume the statement was made for Elman recurrent neural networks, because as far as I know, that is the only type of neural networks for which that statement is valid. Let's say we have an Elman recurrent neural network with one input neuron, one output neuron and one hidden layer with two neurons. In total there are 10 connections. As the image shows, ...


4

This is going backwards, but it kind of follows the logic of the arguments. In terms of efficiency, I can see a few major problems with classical neural networks. Data collection and preprocessing overhead Large neural networks require a lot of data to train. The amount can vary depending on the size of the network and the complexity of the task, but as a ...


4

The basic calculation for a single neuron is of the form $$\sigma\left(\sum_{i} x_i w_i \right),$$ where $x_i$ is the input to the neuron $w_i$ are the neuron-specific weights for every single input and $\sigma$ is the pre-specified activation function. In your terms, and disregarding the activation function, the calculation would turn out to be $$c\,a_c ...


3

Indeed you're intuition is correct, a Boltzmann machine is able to hold more than a Hopfield network in its memory because of its stochastic nature as explored in this paper. In the paper they note that the capacity is around 0.6. After this ratio it starts to break down and adds much more noise to the recalled patterns.


3

Pseudo-random number generators are specifically defined to defeat any form of prediction via 'black box' observation. Certainly, some (e.g. linear congruential) have weaknesses, but you are unlikely to have any success in general in predicting the output of a modern RNG. For devices based on chaotic physical systems (e.g. most national lotteries), there is ...


3

I presume the proof the OP is referring to can be found in this monograph by Hava Siegelmann? In his article 'The Myth of Hypercomputation', the eminent computer scientist Martin Davis explains (p8-9) that there is nothing 'super Turing' about this formulation. EDIT: It's looking like the claim about rational weights being super-Turing is made in this ...


3

It is unclear what kind of network your are referring to, there is not a single neural-network model so conceivable both cases could exist and serve some purpose, yet if you are looking for one that emulates nature and real neurons, then you are missing at least 2 ingredients ( time and the mechanisms of resting potentials and refractory periods), which in ...


3

Both CNN and RNN fall into the super set of neural networks,however applications of the two matters. So to branch them off in terms of applications, I would say CNN’s are mainly used for vision related applications, whereas, RNN’s are mainly used for language processing applications. You can refer to these links for further details. Comparative Study ...


3

CNN vs RNN A CNN will learn to recognize patterns across space while RNN is useful for solving temporal data problems. CNNs have become the go-to method for solving any image data challenge while RNN is used for ideal for text and speech analysis. In a very general way, a CNN will learn to recognize components of an image (e.g., lines, curves, etc.) and ...


3

So the equation that you mentioned is used during the backward pass in which back proppogation is performed in order to make the neural network more accurate. I think you are talking about the state during the forward pass which is completely different. In the forward pass, the neural network is simply run in order to evaluate or it is simply used as a model....


3

It was difficult to find because recurrent network designs predate LSTM extensions of that earlier idea by decades. Although the term recurrent was not yet used as a primary description of the technology advancement, recurrence was an essential feature of the theoretical treatment of artificial networks that learned actions in Attractor dynamics and ...


3

Yes you can apply RNN to any sequence of same data type. The sequence can be in space, time, or any arbitrary ordered list. The items in the sequence can have any data at all, the only requirement is that each represents that same kind of thing (if you have multiple types of thing to process as a sequence, you just need to expand the definition so that the ...


3

As I found this case backs to the sequence labeling. Sequence labeling has some classic solution such as conditional random fields (CRFs) and hidden Markov model (HMM). Also, have some solution in Active learning (AL) which use from algorihtms such as struct SVM ($\text{SVM}^{\text{strcut}}$) like this paper. Also, some NLP solutions in active learning ...


Only top voted, non community-wiki answers of a minimum length are eligible