# Tag Info

15

Your question is quite broad, but here are some tips: For feedforward networks, see this question: @doug's answer has worked for me. There's one additional rule of thumb that helps for supervised learning problems. The upper bound on the number of hidden neurons that won't result in over-fitting is: $$N_h = \frac{N_s} {(\alpha * (N_i + N_o))}$$...

9

A recurrent neural network (RNN) is an artificial neural network that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN). The adjective "recurrent" thus refers to this backward or self-connections, which create loops in these networks. An RNN can be trained using back-...

7

Basically, a CNN saves a set of weights and applies them spatially. For example, in a layer, I could have 32 sets of weights (also called feature maps). Each set of weights is a 3x3 block, meaning I have 3x3x32=288 weights for that layer. If you gave me an input image, for each 3x3 map, I slide it across all the pixels in the image, multiplying the regions ...

7

Select the number of hidden layers and number of memory cells in LSTM is always depend on application domain and context where you want to apply this LSTM. For hidden Layers. The introduction of hidden layer(s) makes it possible for the network to exhibit non-linear behaviour. The optimal number of hidden units could easily be smaller than the number of ...

7

I would recommend to start by reading this blogpost. You can probably cannibalise the code to create a RNN that takes in one statement of a dialogue and then proceeds to output the answer to that statement. That would be the easy version of your project, all without word vectors and thought vectors. You are just inputting characters, so typos don't need to ...

7

An RNN or LSTM have the advantage of "remembering" the past inputs, to improve performance over prediction of a time-series data. If you use a neural network over like the past 500 characters, this may work but the network just treat the data as a bunch of data without any specific indication of time. The network can learn the time representation only ...

6

Recurrent neural networks (RNNs) are artificial neural networks (ANNs) that have one or more recurrent (or cyclic) connections, as opposed to just having feed-forward connections, like a feed-forward neural network (FFNN). These cyclic connections are used to keep track of temporal relations or dependencies between the elements of a sequence. Hence, RNNs ...

6

Recurrent neural networks (RNNs) are a class of artificial neural network architecture inspired by the cyclical connectivity of neurons in the brain. It uses iterative function loops to store information. Difference with traditional Neural networks using pictures from this book: And, an RNN: Notice the difference -- feedforward neural networks' ...

5

The approach you describe is called neuromorphic computing and it's quite a busy field. IBM's TrueNorth even has spiking neurons. The main problem with these projects is that nobody quite knows what to do with them yet. These projects don't try to create chips that are optimised to run a neural network. That would certainly be possible, but the ...

5

RNN's stand for Recurrent Neural Networks which is, in fact, Deep Learning. There has to be a loss since you're dealing with supervised learning and the typical loss metrics used are the same as you would see in feedforward networks (usually binary cross-entropy), the main difference being loss would be calculated between the true label at a particular time ...

4

If neurons and synapses can be implemented using transistors, I hope you are not talking about the neural networks which are currently winning all competitions in machine learning (MLPs, CNNs, RNNs, Deep Residual Networks, ...). Those were once used as a model for neurons, but they are only very loosely related to what happens in real brain cells. Spiking ...

4

I assume the statement was made for Elman recurrent neural networks, because as far as I know, that is the only type of neural networks for which that statement is valid. Let's say we have an Elman recurrent neural network with one input neuron, one output neuron and one hidden layer with two neurons. In total there are 10 connections. As the image shows, ...

4

You are right. I think you are just misinterpreting the part of the sentence ('specifically LSTMs'). LSTMs are an example of a popular type of RNN. RNNs and CNNs are different architectures but they can be used together. Here is another sentence with the same structure: It is clear than dogs, specifically corgis, and cats are very common in online memes.

4

In general, there are no guidelines on how to determine the number of layers or the number of memory cells in a LSTM. The number of layers and cells required in an LSTM might depend on several aspects of the problem: The complexity of the dataset. The amount of features, number of data points etc. The the data generating process. Following example of how ...

4

I think there are two parts to answering this question. First, about the specific paper that has been mentioned. The paper's title is hyperbolic, and probably written that way to get more people to read it. The paper itself does not make the claim that attention-based networks will supplant existing recurrent network architectures. Instead, it makes a more ...

4

The basic calculation for a single neuron is of the form $$\sigma\left(\sum_{i} x_i w_i \right),$$ where $x_i$ is the input to the neuron $w_i$ are the neuron-specific weights for every single input and $\sigma$ is the pre-specified activation function. In your terms, and disregarding the activation function, the calculation would turn out to be $$c\,a_c ... 3 Both CNN and RNN fall into the super set of neural networks,however applications of the two matters. So to branch them off in terms of applications, I would say CNN’s are mainly used for vision related applications, whereas, RNN’s are mainly used for language processing applications. You can refer to these links for further details. Comparative Study ... 3 I presume the proof the OP is referring to can be found in this monograph by Hava Siegelmann? In his article 'The Myth of Hypercomputation', the eminent computer scientist Martin Davis explains (p8-9) that there is nothing 'super Turing' about this formulation. EDIT: It's looking like the claim about rational weights being super-Turing is made in this ... 3 It is unclear what kind of network your are referring to, there is not a single neural-network model so conceivable both cases could exist and serve some purpose, yet if you are looking for one that emulates nature and real neurons, then you are missing at least 2 ingredients ( time and the mechanisms of resting potentials and refractory periods), which in ... 3 Pseudo-random number generators are specifically defined to defeat any form of prediction via 'black box' observation. Certainly, some (e.g. linear congruential) have weaknesses, but you are unlikely to have any success in general in predicting the output of a modern RNG. For devices based on chaotic physical systems (e.g. most national lotteries), there is ... 3 It was difficult to find because recurrent network designs predate LSTM extensions of that earlier idea by decades. Although the term recurrent was not yet used as a primary description of the technology advancement, recurrence was an essential feature of the theoretical treatment of artificial networks that learned actions in Attractor dynamics and ... 3 Yes you can apply RNN to any sequence of same data type. The sequence can be in space, time, or any arbitrary ordered list. The items in the sequence can have any data at all, the only requirement is that each represents that same kind of thing (if you have multiple types of thing to process as a sequence, you just need to expand the definition so that the ... 3 First, you need to consider what are the "parameters" of this "optimization algorithm" that you want to "optimize". Let's take the most simple case, a SGD without momentum. The update rule for this optimizer is:$$ w_{t+1} \leftarrow w_{t} - a \cdot \nabla_{w_{t}} J(w_t) = w_{t} - a \cdot g_t  where $w_t$ are the weights at iteration $t$, $J$ is the cost ...

2

Indeed you're intuition is correct, a Boltzmann machine is able to hold more than a Hopfield network in its memory because of its stochastic nature as explored in this paper. In the paper they note that the capacity is around 0.6. After this ratio it starts to break down and adds much more noise to the recalled patterns.

2

There is nothing wrong in theory. It is reasonable to expect that for some problems it will work worse than taking the immediate answer on the last known part of the sequence, some it will have roughly the same effect, and for others it might result in a better result. A trained RNN is essentially a learned arbitrary program. If a computer program that ...

2

The best option in your case would probably be zero padding or padding up. This is simply zeroing out inputs for cases in which there is no data. It's done a lot on the borders of images for CNNs. That or you could just use a RNN which can handle your variable length inputs with ease.

2

So the equation that you mentioned is used during the backward pass in which back proppogation is performed in order to make the neural network more accurate. I think you are talking about the state during the forward pass which is completely different. In the forward pass, the neural network is simply run in order to evaluate or it is simply used as a model....

2

Maybe you should have a look at this: https://wiki.inf.ed.ac.uk/twiki/pub/CSTR/ListenTerm1201415/sak2.pdf Here they show that 2 layers are nice, 5 layers are better and 7 layers are very hard to train.

2

I haven't seen any dataset where some standard models worked and neural networks utterly failed. For columnar data (e.g. Excel files / database dumps / CSV files) which contain structured data usually tree-based models like random forests and gradient boosting work better, but neural networks are also usually way better than random. If you demand other ...

Only top voted, non community-wiki answers of a minimum length are eligible