29

I'll list some bullet points of the main innovations introduced by transformers , followed by bullet points of the main characteristics of the other architectures you mentioned, so we can then compared them. Transformers Transformes (Attention is all you need) were introduced in the context of machine translation with the purpose to avoid recursion in order ...


25

Your question is quite broad, but here are some tips. Specifically for LSTMs, see this Reddit discussion Does the number of layers in an LSTM network affect its ability to remember long patterns? The main point is that there is usually no rule for the number of hidden nodes you should use, it is something you have to figure out for each case by trial and ...


13

This is a question of time series forecasting, since your numbers form a sequence. You may want to take a look at the "forecasting" tag at CrossValidated. If you have only 700 data points, ML/AI methods will likely not be very useful. Whatever you do, I would recommend you benchmark your chosen method against very simple approaches, like the ...


11

The two tech reports below both call RNNs explicitly "recurrent net(work)s". Rumelhart, David E; Hinton, Geoffrey E, and Williams, Ronald J (Sept. 1985). Learning internal representations by error propagation. Tech. rep. ICS 8504. San Diego, California: Institute for Cognitive Science, University of California. Jordan, Michael I. (May 1986). ...


10

Actually, there is usually an upper bound for inputs of transformers, due to the inability of handling long-sequence. Usually, the value is set as 512 or 1024 at current stage. However, if you are asking handling the various input size, adding padding token such as [PAD] in BERT model is a common solution. The position of [PAD] token could be masked in self-...


10

The Wikipedia article is more technically correct, in that the term RNN is formally taken to mean "a neural network with recurrent connections", and that includes many architectures that match this description, including LSTMs. However, it is also common to see "RNN" used as a short-hand for a kind of "Vanilla RNN" or "...


9

A recurrent neural network (RNN) is an artificial neural network that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN). The adjective "recurrent" thus refers to this backward or self-connections, which create loops in these networks. An RNN can be trained using back-...


8

I would recommend to start by reading this blogpost. You can probably cannibalise the code to create a RNN that takes in one statement of a dialogue and then proceeds to output the answer to that statement. That would be the easy version of your project, all without word vectors and thought vectors. You are just inputting characters, so typos don't need to ...


8

The selection of the number of hidden layers and the number of memory cells in LSTM probably depends on the application domain and context where you want to apply this LSTM. The optimal number of hidden units could be smaller than the number of inputs. AFAIK, there is no rule like multiply the number of inputs with $N$. If you have a lot of training ...


8

As all you have is a series of numbers, you should try using a sequence model. I suggest you look into RNNs and in particular LSTMs. Of course this is assuming despite the lack of "obvious patterns", there are some kind of hidden patterns in your data. If not, what you have is not very different than random walk in 3 dimensions - which makes the ...


7

Recurrent neural networks (RNNs) are artificial neural networks (ANNs) that have one or more recurrent (or cyclic) connections, as opposed to just having feed-forward connections, like a feed-forward neural network (FFNN). These cyclic connections are used to keep track of temporal relations or dependencies between the elements of a sequence. Hence, RNNs ...


7

Basically, a CNN saves a set of weights and applies them spatially. For example, in a layer, I could have 32 sets of weights (also called feature maps). Each set of weights is a 3x3 block, meaning I have 3x3x32=288 weights for that layer. If you gave me an input image, for each 3x3 map, I slide it across all the pixels in the image, multiplying the regions ...


7

I think there are two parts to answering this question. First, about the specific paper that has been mentioned. The paper's title is hyperbolic, and probably written that way to get more people to read it. The paper itself does not make the claim that attention-based networks will supplant existing recurrent network architectures. Instead, it makes a more ...


7

Recurrent neural networks (RNNs) are a class of artificial neural network architecture inspired by the cyclical connectivity of neurons in the brain. It uses iterative function loops to store information. Difference with traditional Neural networks using pictures from this book: And, an RNN: Notice the difference -- feedforward neural networks' ...


7

An RNN or LSTM have the advantage of "remembering" the past inputs, to improve performance over prediction of a time-series data. If you use a neural network over like the past 500 characters, this may work but the network just treat the data as a bunch of data without any specific indication of time. The network can learn the time representation only ...


6

There is a technique called Pruning in neural networks, which is used just for this same purpose. The pruning is done on the number of hidden layers. The process is very similar to the pruning process of decision trees. The pruning process is done as follows: Train a large, densely connected, network with a standard training algorithm Examine the trained ...


6

In general, there are no guidelines on how to determine the number of layers or the number of memory cells in an LSTM. The number of layers and cells required in an LSTM might depend on several aspects of the problem: The complexity of the dataset, such as the number of features, the number of data points, etc. The data-generating process. For example, ...


6

The approach you describe is called neuromorphic computing and it's quite a busy field. IBM's TrueNorth even has spiking neurons. The main problem with these projects is that nobody quite knows what to do with them yet. These projects don't try to create chips that are optimised to run a neural network. That would certainly be possible, but the ...


6

You are talking about two different types of 'size'. The size of the input for a FFNN and a RNN must always remain fixed for the same network architecture, i.e. they take in a vector $x \in \mathbb{R}^d$ and could not take as input for instance a vector $y \in \mathbb{R}^b$ where $b \neq d$. The size you refer to in the context of the RNN is the length of ...


6

I guess the most "suitable" approach is to look up research papers on ML/AI/Stats based methods on bipolar disorder mood swings prediction/regression etc. Focus on the abstract, intro/related works and conclusion. Find out why the method is proposed, what the well-known approaches are, what the intuition for the proposed methods are. Find out the ...


5

This is going backwards, but it kind of follows the logic of the arguments. In terms of efficiency, I can see a few major problems with classical neural networks. Data collection and preprocessing overhead Large neural networks require a lot of data to train. The amount can vary depending on the size of the network and the complexity of the task, but as a ...


5

This is my own understanding of hidden state in a recurrent network and if its wrong please feel free to let me know. Lets take this simple sequence first, X = [a,b,c,d,.......,y,z] Y = [b,c,d,e,.......,z,a] Instead of RNN we will first try to train this in a simple multi layer neural network with one input and one output, here hidden layers details ...


5

RNN's stand for Recurrent Neural Networks which is, in fact, Deep Learning. There has to be a loss since you're dealing with supervised learning and the typical loss metrics used are the same as you would see in feedforward networks (usually binary cross-entropy), the main difference being loss would be calculated between the true label at a particular time ...


5

Let's start with RNN. A well known problem is vanishin/exploding gradients, which means that the model is biased by most recent inputs in the sequence, or in other words, older inputs have practically no effect in the output at the current step. LSTMs/GRUs mainly try to solve this problem, by including a separate memory (cell) and/or extra gates to learn ...


5

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas parameters is more often found in statistics literature). Batch size, learning rate etc. are hyper-parameters which basically means they are user specified, whereas ...


4

If neurons and synapses can be implemented using transistors, I hope you are not talking about the neural networks which are currently winning all competitions in machine learning (MLPs, CNNs, RNNs, Deep Residual Networks, ...). Those were once used as a model for neurons, but they are only very loosely related to what happens in real brain cells. Spiking ...


4

I assume the statement was made for Elman recurrent neural networks, because as far as I know, that is the only type of neural networks for which that statement is valid. Let's say we have an Elman recurrent neural network with one input neuron, one output neuron and one hidden layer with two neurons. In total there are 10 connections. As the image shows, ...


4

Have a look at the paper Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling (2014), where different LSTM architectures are compared. In the abstract, the authors write the following. We show that a two-layer deep LSTM RNN where each LSTM layer has a linear recurrent projection layer can exceed state-of-the-art ...


4

You are right. I think you are just misinterpreting the part of the sentence ('specifically LSTMs'). LSTMs are an example of a popular type of RNN. RNNs and CNNs are different architectures but they can be used together. Here is another sentence with the same structure: It is clear than dogs, specifically corgis, and cats are very common in online memes.


4

The basic calculation for a single neuron is of the form $$\sigma\left(\sum_{i} x_i w_i \right),$$ where $x_i$ is the input to the neuron $w_i$ are the neuron-specific weights for every single input and $\sigma$ is the pre-specified activation function. In your terms, and disregarding the activation function, the calculation would turn out to be $$c\,a_c ...


Only top voted, non community-wiki answers of a minimum length are eligible