9

A recurrent neural network (RNN) is an artificial neural network that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN). The adjective "recurrent" thus refers to this backward or self-connections, which create loops in these networks. An RNN can be trained using back-...


7

An RNN or LSTM have the advantage of "remembering" the past inputs, to improve performance over prediction of a time-series data. If you use a neural network over like the past 500 characters, this may work but the network just treat the data as a bunch of data without any specific indication of time. The network can learn the time representation only ...


7

There is a technique called Pruning in neural networks, which is used just for this same purpose. The pruning is done on the number of hidden layers. The process is very similar to the pruning process of decision trees. The pruning process is done as follows: Train a large, densely connected, network with a standard training algorithm Examine the trained ...


7

Let's suppose that we have an MLP with $15$ inputs, $20$ hidden neurons and $2$ output neurons. The operations performed are only in the hidden and output neurons, given that the input neurons only represent the inputs (so they do not perform any operation). Each hidden neuron performs a linear combination of its inputs followed by the application of a non-...


6

Recurrent neural networks (RNNs) are a class of artificial neural network architecture inspired by the cyclical connectivity of neurons in the brain. It uses iterative function loops to store information. Difference with traditional Neural networks using pictures from this book: And, an RNN: Notice the difference -- feedforward neural networks' ...


6

A layer with bigger number of nodes than previous one is something very common. Some examples are: strategies encoder-decoder (autoencoders) where the encoder typically has layers with a decreasing number of nodes (until the compressed/encoded data) and the decoder has layers increasing in number of nodes. bidirectional recurrent networks where in the ...


5

You described a single-layer feedforward network. They can have multiple layers. The significance of the weights is that they make a linear transformation from the output of the previous layer and hand it to the node they are going to. To say it more simplistically, they specify how important (and in what way: negative or positive) is the activation of node ...


4

Actually, this already exists! I happened to make a presentation of a paper that talks about this topic. These networks are called DenseNets, which stands for densely connected convolutional networks. Just like in your question, within a dense block, the output of each layer is given as input to all subsequent layers. Put another way, in a normal feed-...


3

In reallity any continous function on a compact can be approximated by a neural network having one hidden layer with a finite number of neurones (This is the Universal Approximation Theorem). Thus you only need one hidden layer to approximate the multiplication on a compact, note that you need to apply a non linear activation on the hidden layer to do this.


3

The Project Summarized The project goal appears to be a common one: Routing correspondence in an efficient manner to maintain good but low cost customer and public relations. A few features of the project were mentioned. Neural network project Received some design and project history from predecessor Classifies messages for telcos Sends results to support ...


3

You need to scale the input. Neural networks work best with a limited input domain, and train badly when it is exceeded. For statistical data, you would typically scale your input to have mean 0, standard deviation 1. Here, you will be better off fitting the input to roughly -1 to 1. Up to you where you scale the values, but usually this is done outside ...


3

tl;dr The equivalent to a neuron in a Fully-Connected (FC) layer is the kernel (or filter) of a Convolution layer Differences The neurons of these two types of layers have two key differences. These are that the convolution layers implement: Sparse connectivity, i.e. each neuron is connected only to an area of the input, not the whole. Weight sharing, i.e....


3

Introduction First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't completely answer the original question. In fact, I leave some unanswered questions at the end (that I will eventually answer). The gradient is a linear ...


3

A simple feed-forward neural network with at least one hidden layer would suffice in your problem, and can deal with arbitrary non-linear relationships between input and output. If you expect relationships to be highly non-linear then additional layers might be required, but from your description of the problem, I would be surprised if you needed more than ...


2

It is correct that climate and economic models are distinct from waste models. A memory based model is the correct approach because the time domain is key in prediction based on existing trend data. However, the RNN is not a practical production model with others that have proven more productive. Although this article is trendy and not accurate in every ...


2

In the paper The Limitations of Deep Learning in Adversarial Settings, Papernot et. al., 2016, IEEE the chain rule is used, "To express $\nabla F(X∗)$ in terms of $X$ and constant values only." Earlier is stated, "Our understanding of how changes made to inputs affect a DNN’s output stems from the evaluation of the forward derivative: a matrix we introduce ...


2

What I notice is that the network's fitness keeps climbing up and falling down again. It seems that my current approach only evolves certain patterns on placing signs on the board and once random mutation interrupts current pattern new one emerges. My network goes in circles without ever evolving actual strategy. I suspect solution for this would be to pit ...


2

First your questions: Yes, it is possible to accomplish this with a neural network, this is actually very similar to your current working model (the idea is the same, just different classes). So, there are no real reasons to start changing the architecture, especially if from what I understand, you are not a deep-learning expert. I mean, I'm sure that you ...


2

You can not use the ready code directly without any manipulation. Because every piece of code is written for specific datasets. In the article that you mentioned, the writer created a small dataset and then he creates an ANN architecture for it. If you want to build an ANN based on Iris dataset, you should think and create an architecture on paper maybe ...


2

Assumptions Different model structures encode different assumptions - while we often make simplifying assumptions that aren't strictly correct, some assumptions are more wrong than others. For example, your proposed structure of "just pass the $X$ number of letters leading up to the last letter into an FFNN" makes an assumption that all the information ...


2

Neural networks (NNs) are usually classified into feed-forward (i.e. NNs with feedforward connections), recurrent (i.e. NNs with recurrent connections) and convolutional (i.e. NNs that perform a convolution or cross-correlation operation). The term multi-layer NN may also be used to refer to feed-forward neural networks or, in general, neural networks with ...


2

Such a network could be either a Residual Network or a Highway Network depending upon the underlying architecture of the skip layers. They are primarily used to to tackle the problem of vanishing gradients in very deep networks by reusing activations from a previous layer and passing them to adjacent layers (two or three skips away). Highway Network: (...


2

This could be called a residual neural network (ResNet), which is a neural network with skip connections, that is, connections that skip layers. Here's a screenshot of a figure from the paper Deep Residual Learning for Image Recognition (2015), an important paper that shows the usefulness of these architectures.


2

It is common during the training of Neural Networks for accuracy to improve for a while and then get worse -- in general, This is caused by over-fitting. It's also fairly common for the Neural Network to "get UNLUCKY and get knocked into a BAD sectors of parameter space corresponding to a sudden decrease in accuracy -- sometimes it can recover from this ...


2

Q learning on its own isn't enough to learn a winning strategy for a game like 2048. 2048 requires predictive thinking for possible outcomes and good positional awareness. Performance of the agent is heavily dependent on the reward function. The approach to give the reward proportional to the obtained points after every move is naive since it might sacrifice ...


2

Assume the image can contain objects of class $C_1 \dots C_c$. Assume a set of additional inputs that has a meaning of questions as "contains the image a C_i or C_j or ... ?". The main problem for the system is classify the image in classes $C_i$. Second problem is answer the implicit question proposed by the remainder inputs. Thus, better combine ...


2

The main benefit of deep learning is that you don't have to manually design features. Classic Machine Learning algorithms always include the Feature engineering step, whereas neural networks are able to crate features automatically during learning. The classic example is CNN. In the first layer it creates simple features that representing lines, the last ...


1

TL;DR: You'll need to store a little bit more to perform backward passes. You'll need to store data from the forward pass. This stored information is used for calculating the gradient. Overview (warning: not trivial) I know the weights can just be stored in an array You'll need a little more: To update the weights you need to keep a "cache" of the ...


1

Have a look at and read the paper Playing Atari with Deep Reinforcement Learning, which describes deep Q-learning (i.e. Q-learning with neural networks). In particular, have a look at algorithm 1 (on page 5). As it is usually the case in deep learning, gradient descent and back-propagation are used to update the parameters (or weights) of the neural network, ...


1

In an RNN, the output of the previous state is passed as an input to the current state. Intuitively, there is a temporal (time-based) relationship in the way in which input is processed in an RNN. It can understand how the current state was achieved on the basis of the previous values, i.e value at time-step $t$ is a result of value at time-steps $t-1, t-2$, ...


Only top voted, non community-wiki answers of a minimum length are eligible