11

Formally, a single hidden layer is sufficient to approximate a continuous function to any desired degree of accuracy, so in that sense, you never need more than 1. This is called the Universal Approximation Theorem. Finding the best topology for a given problem is an open research problem. As far as I know, there are few universal 'rules of thumb' for this. ...


10

It is suggested that the number of hidden units in a layer should be in powers of 2 because it helps converge faster. I would quite like to see a reference to this suggestion, in case it has been misunderstood. As far as I know, there is no such effect in normal neural networks. In convolutional neural networks it might potentially be true in a minor way ...


7

There is a technique called Pruning in neural networks, which is used just for this same purpose. The pruning is done on the number of hidden layers. The process is very similar to the pruning process of decision trees. The pruning process is done as follows: Train a large, densely connected, network with a standard training algorithm Examine the trained ...


6

I have an idea to find the optimal number of hidden neurons required in a neural network but I'm not sure how accurate it is. It's a complete non-starter, and there is a no such calculation possible in the general case (real-valued inputs to a neural network). Even with one input neuron it is not possible. That is because even with one input, the output ...


6

A layer with bigger number of nodes than previous one is something very common. Some examples are: strategies encoder-decoder (autoencoders) where the encoder typically has layers with a decreasing number of nodes (until the compressed/encoded data) and the decoder has layers increasing in number of nodes. bidirectional recurrent networks where in the ...


5

Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e layer_output = Weights.layer_input + bias Consider a 2 layer neural network, the outputs from layer one will be: x2 = W1*x1 + b1 Now we pass the same input to the second layer, which will be x3 = W2x*2 + b2 Also x2 = W1*x1 + b1 Substituting ...


5

"Hidden" layers really aren't all that special... a hidden layer is really no more than any layer that isn't input or output. So even a very simple 3 layer NN has 1 hidden layer. So I think the question isn't really "How do hidden layers help?" as much as "Why are deeper networks better?". And the answer to that latter question is an area of active ...


4

I assume the statement was made for Elman recurrent neural networks, because as far as I know, that is the only type of neural networks for which that statement is valid. Let's say we have an Elman recurrent neural network with one input neuron, one output neuron and one hidden layer with two neurons. In total there are 10 connections. As the image shows, ...


4

Actually, this already exists! I happened to make a presentation of a paper that talks about this topic. These networks are called DenseNets, which stands for densely connected convolutional networks. Just like in your question, within a dense block, the output of each layer is given as input to all subsequent layers. Put another way, in a normal feed-...


3

It depends on the accuracy you want. If you had 1 neuron, it could discern things across a line, if you have 2, you could solve things across 2 lines, etc. As you increase the number of neurons, you are increasing the number of discernible areas. As you increase the number of lines you can use to break up the input space, the lines can be placed to ...


3

Sigmoid > Hyperbolic tangent: As you mentioned, the application of Sigmoid might be more convenient than hyperbolic tangent in the cases that we need a probability value at the output (as @matthew-graves says, we can fix this with a simple mapping/calibration step). In other layers, this makes no sense. Hyperbolic tangent > Sigmoid: Hyperbolic tangent has ...


3

I don't think it makes sense to decide activation functions based on desired properties of the output; you can easily insert a calibration step that maps the 'neural network score' to whatever units you actually want to use (dollars, probability, etc.). So I think preference between different activation functions mostly boils down to the different ...


3

Your code suggests a likely problem here: It looks like you are training a very deep neural network with sigmoidal activation functions at every layer. The sigmoid has the property that its derivative (S*(1-S)) will be extremely small when the activation function's value is close to 0 or close to 1. In fact, the largest it can be is about 0.25. The ...


3

You are partially correct. On CNNs the output shape per layer is defined by the amount of filters used, and the application of the filters (dilation, stride, padding, etc.). CNNs shapes In your example, your input is 30 x 30 x 3. Assuming stride of 1, no padding, and no dilation on the filter, you will get a spatial shape equal to your input, that is ...


3

For a 3 channel image (RGB), each filter in a convolutional layer computes a feature map which is essentially a single channel image. Typically, 2D convolutional filters are used for multichannel images. This can be a single filter applied to each layer or a seperate filter per layer. These filters are looking for features which are independent of the color,...


3

Hi and welcome to the community. It's important to understand these basic concepts very clearly. You have to first understand the basic unit of a neural network, a single node/neuron/perceptron. Let us forget all about Neural Networks for a bit, and talk about something far simpler. Linear Regression In the above figure, we clearly have one independent ...


2

Hidden layers by themselves aren't useful. If you had hidden layers that were linear, the end result would still be a linear function of the inputs, and so you could collapse an arbitrary number of linear layers down to a single layer. This is why we use nonlinear activation functions, like RELU. This allows us to add a level of nonlinear complexity with ...


2

I recommend you read up on reinforcement learning. Seeing how AirHockey is similar to the old Atari game Pong, here is a write-up (with code) about how to implement a simple neural network, that plays the game; Deep Reinforcement Learning: Pong from Pixels. As far as the choice of layers go, again, you should read up on deep reinforcement learning, perhaps ...


2

There are many problems requiring more than two hidden layers. Randomly select a recent Google journal paper on deep learning, you'll see their network could have something like 5 (or more) hidden layers. Justin Domke wrote his notes for students, so he probably tried to make his points as simple as possible. For a "typical" machine learning problem that ...


2

Yes, your understanding of the hidden state is correct. But the size of the hidden state is a hyperparameter that needs to found by trial-and-error. There is no closed-form formula or solution which links the size of the hidden state and the problem at hand. But, there are some rules of thumb like to start out with the size of the hidden state to be a power ...


2

I believe, this post and this post written by me well address your question. EDIT In fact, this is a very interesting question that you ask. It deserves some more explanation. 1. Fully connected networks The more layers you add, the more "nonlinear" your network becomes. For instance, in the case of two spirals problem, which requires a "highly nonlinear ...


2

By itself, I'm not sure it's possible to know. It's possible the slides were old. Or, the intended purpose was to mention how as sigmoid ranges from 0 to 1. Mostly, it looks like it was intended to bring up gradient descent. But it could also be an entry point to the discussion of other methods such as ReLU. Either that or perhaps some sort of norming ...


2

It's depend more on number of classes. For 20 classes 2 layers 512 should be more then enough. If you want to experiment you can try also 2 x 256 and 2 x 1024. Less then 256 may work too, but you may underutilize power of previous conv layers.


2

This is my own understanding of hidden state in a recurrent network and if its wrong please feel free to let me know. Lets take this simple sequence first, X = [a,b,c,d,.......,y,z] Y = [b,c,d,e,.......,z,a] Instead of RNN we will first try to train this in a simple multi layer neural network with one input and one output, here hidden layers details ...


2

About the images inside the CNN layers: I really recommend this article since there is no one short answer to this question and it probably will be better to experiment with it. About the RGB input images: When needed to train on RGB pictures it is not advised to split the RGB channels, you can think of it by trying to identify a fictional cat with red ears,...


2

From here: Using other activation functions don’t provide significant improvement in performance and tweaking them doesn’t provide any big improvement. So as per simplicity we use same activation function for most of the case in Deep Neural Networks.


2

You probably got the back propagation wrong. I have done a test on the accuracy on adding an extra layer and the accuracy went up from 94% to 96% for me. See this for details: https://colab.research.google.com/drive/17kAJ2KJ36grG9sz-KW10fZCQW9i2Tf2c To run the notebook click Open in playground and run the code. There is a commented line which add 1 extra ...


1

I guess the reason is that we call them nodes and nodes are usually depicted using circles in graph theory. We model them as graphs for both forward and backward paths.


1

The information you are probably missing is that word embeddings are learned on the basis of context. For example, you might try to predict a vector for a word from the wordvectors of the other words in the same sentence. This way word vectors of words that occur in similar contexts will turn out to be similar. You can think of it as word vectors not ...


1

The subword-based embedding is rather visual and easily understandable. However, the autoencoder embedding is what machines understand the componential meaning of words. 1) An autoencoder embedding layer can be trained together with other layers to fit with the relation of data in dataset. 2) Or the embedding layer can be kept unchanged as used as a ...


Only top voted, non community-wiki answers of a minimum length are eligible