# Tag Info

11

Formally, a single hidden layer is sufficient to approximate a continuous function to any desired degree of accuracy, so in that sense, you never need more than 1. This is called the Universal Approximation Theorem. Finding the best topology for a given problem is an open research problem. As far as I know, there are few universal 'rules of thumb' for this. ...

8

There is a technique called Pruning in neural networks, which is used just for this same purpose. The pruning is done on the number of hidden layers. The process is very similar to the pruning process of decision trees. The pruning process is done as follows: Train a large, densely connected, network with a standard training algorithm Examine the trained ...

6

It is suggested that the number of hidden units in a layer should be in powers of 2 because it helps converge faster. I would quite like to see a reference to this suggestion, in case it has been misunderstood. As far as I know, there is no such effect in normal neural networks. In convolutional neural networks it might potentially be true in a minor way ...

6

I have an idea to find the optimal number of hidden neurons required in a neural network but I'm not sure how accurate it is. It's a complete non-starter, and there is a no such calculation possible in the general case (real-valued inputs to a neural network). Even with one input neuron it is not possible. That is because even with one input, the output ...

5

Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e layer_output = Weights.layer_input + bias Consider a 2 layer neural network, the outputs from layer one will be: x2 = W1*x1 + b1 Now we pass the same input to the second layer, which will be x3 = W2x*2 + b2 Also x2 = W1*x1 + b1 Substituting ...

5

"Hidden" layers really aren't all that special... a hidden layer is really no more than any layer that isn't input or output. So even a very simple 3 layer NN has 1 hidden layer. So I think the question isn't really "How do hidden layers help?" as much as "Why are deeper networks better?". And the answer to that latter question is an area of active ...

4

I assume the statement was made for Elman recurrent neural networks, because as far as I know, that is the only type of neural networks for which that statement is valid. Let's say we have an Elman recurrent neural network with one input neuron, one output neuron and one hidden layer with two neurons. In total there are 10 connections. As the image shows, ...

3

Actually it doeas! I happened to make a presentation of a paper that talks about this topic. They called it DenseNet, which stands for densely connected convolutional networks. Just like in your question, within a denseblock the output of each layer is given as input to all subsequent layers. Put another way, in a normal feed forward neural network the l-th ...

3

It depends on the accuracy you want. If you had 1 neuron, it could discern things across a line, if you have 2, you could solve things across 2 lines, etc. As you increase the number of neurons, you are increasing the number of discernible areas. As you increase the number of lines you can use to break up the input space, the lines can be placed to ...

3

I don't think it makes sense to decide activation functions based on desired properties of the output; you can easily insert a calibration step that maps the 'neural network score' to whatever units you actually want to use (dollars, probability, etc.). So I think preference between different activation functions mostly boils down to the different ...

2

Sigmoid > Hyperbolic tangent: As you mentioned, the application of Sigmoid might be more convenient than hyperbolic tangent in the cases that we need a probability value at the output (as @matthew-graves says, we can fix this with a simple mapping/calibration step). In other layers, this makes no sense. Hyperbolic tangent > Sigmoid: Hyperbolic tangent has ...

2

Hidden layers by themselves aren't useful. If you had hidden layers that were linear, the end result would still be a linear function of the inputs, and so you could collapse an arbitrary number of linear layers down to a single layer. This is why we use nonlinear activation functions, like RELU. This allows us to add a level of nonlinear complexity with ...

2

I recommend you read up on reinforcement learning. Seeing how AirHockey is similar to the old Atari game Pong, here is a write-up (with code) about how to implement a simple neural network, that plays the game; Deep Reinforcement Learning: Pong from Pixels. As far as the choice of layers go, again, you should read up on deep reinforcement learning, perhaps ...

2

I believe, this post and this post written by me well address your question. EDIT In fact, this is a very interesting question that you ask. It deserves some more explanation. 1. Fully connected networks The more layers you add, the more "nonlinear" your network becomes. For instance, in the case of two spirals problem, which requires a "highly nonlinear ...

2

By itself, I'm not sure it's possible to know. It's possible the slides were old. Or, the intended purpose was to mention how as sigmoid ranges from 0 to 1. Mostly, it looks like it was intended to bring up gradient descent. But it could also be an entry point to the discussion of other methods such as ReLU. Either that or perhaps some sort of norming ...

2

It's depend more on number of classes. For 20 classes 2 layers 512 should be more then enough. If you want to experiment you can try also 2 x 256 and 2 x 1024. Less then 256 may work too, but you may underutilize power of previous conv layers.

2

Your code suggests a likely problem here: It looks like you are training a very deep neural network with sigmoidal activation functions at every layer. The sigmoid has the property that its derivative (S*(1-S)) will be extremely small when the activation function's value is close to 0 or close to 1. In fact, the largest it can be is about 0.25. The ...

1

This is my own understanding of hidden state in a recurrent network and if its wrong please feel free to let me know. Lets take this simple sequence first, X = [a,b,c,d,.......,y,z] Y = [b,c,d,e,.......,z,a] Instead of RNN we will first try to train this in a simple multi layer neural network with one input and one output, here hidden layers details ...

1

I guess the reason is that we call them nodes and nodes are usually depicted using circles in graph theory. We model them as graphs for both forward and backward paths.

1

The information you are probably missing is that word embeddings are learned on the basis of context. For example, you might try to predict a vector for a word from the wordvectors of the other words in the same sentence. This way word vectors of words that occur in similar contexts will turn out to be similar. You can think of it as word vectors not ...

1

The subword-based embedding is rather visual and easily understandable. However, the autoencoder embedding is what machines understand the componential meaning of words. 1) An autoencoder embedding layer can be trained together with other layers to fit with the relation of data in dataset. 2) Or the embedding layer can be kept unchanged as used as a ...

1

You need to perform Hyperparameter Tuning to identify - Number of hidden layers. Number of neurons in each of the hidden layers. Dropout The activation function you use in each of your hidden layers. There parameters are only related to how you build your model. There are others that relate to training like batch size, number of epochs and so on. Your ...

1

First of all, when you add hidden layers, or stack RBMs, you get a Deep Belief Network (DBN). Your question then deals with the comparison of DBNs and RBMs. There are some elements to answer this question in the article Representational Power of Restricted Boltzmann Machines and Deep Belief Networks by Nicolas Le Roux, which can be found summarized in these ...

1

It was proven a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions (see Universal approximation theorem). More layers can't improve something that can already do "everything". But adding more layers reduces the number of necessary neurons, and reduces computing power needed for the ...

1

More hidden layers will just escalate the possibilities amount the neurons, including the solutions from the previous hidden layers. (I will edit this once I am at home and provide you with a good link I found some time ago) Meanwhile maybe this will help you https://stats.stackexchange.com/questions/63152/what-does-the-hidden-layer-in-a-neural-network-...

1

Welcome to the world of deep learning! Yes, your understanding of the hidden state is correct. But the size of the hidden state is a hyperparameter that needs to found by trial-and-error. There is no closed-form formula or solution which links the size of the hidden state and the problem at hand. But, there are some rules of thumb like to start out with the ...

1

When we add a single layer with a non-linear activation function, right after the application of activation function, a new basis function is found(for that neuron) which is some combination of weights and biases, which acts as a new way to view or analyse the feature sets. With increasingly deep network, we keep finding representations which are new basis ...

1

In general no. The research area that you seem to be looking for is called generalization, which is very much so still an active area of research. Actual architecture design strongly depends on the dataset itself and available resources, and thus there isn't a general rule of thumb that works for every case.

1

I also recommend you take a look at the following work by Uber AI Labs who used an interesting approach to computer games: https://eng.uber.com/deep-neuroevolution/

1

There are many problems requiring more than two hidden layers. Randomly select a recent Google journal paper on deep learning, you'll see their network could have something like 5 (or more) hidden layers. Justin Domke wrote his notes for students, so he probably tried to make his points as simple as possible. For a "typical" machine learning problem that ...

Only top voted, non community-wiki answers of a minimum length are eligible