Questions tagged [hidden-layers]

For questions about functioning, applications, structure and improvement of performance of hidden layers in a Neural Network.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

Are there guidelines or rules of thumb on how to stack hidden layers in a RNN?

I’m currently working on the prediction of chaotic data and I have decided to see how well would an RNN, namely an LSTM, would do. I am fairly new to the topic of Neural Networks, but I have found a ...
Jxson99's user avatar
  • 101
0 votes
2 answers
50 views

How are hidden layers counted / semantically defined?

I'm working my way through how LLMs work and I understand how things work but it's not clear to me exactly what is semantically defined as a "layer". Using the following FFN as an example: ...
Grant Curell's user avatar
0 votes
0 answers
11 views

How are groups created in maxout units when dividing the set of inputs 𝑧 into groups of 𝑘 values?

I don't get $G^(i)$the set of indices into the inputs for group $i$, $\{(i −1)k+ 1, . . . , ik\}$ when creating a maxout units/function, these thing that outputs the maximum element of groups: $$g(z)...
Revolucion for Monica's user avatar
2 votes
2 answers
50 views

Are there any toy classification problems that can't be solved with logistic regression, but can be solved with a NN with exactly one hidden node?

Basically, I'm wondering if there are any small and simple problems that are: complex enough to be unsolvable with a standard neural network without any hidden layer (ie. input -> output) simple ...
J Doug's user avatar
  • 23
0 votes
0 answers
40 views

Is ANN architecture mesh topology exist?

I'm just wondering if there's ANN architecture that looks like mesh topology at context of computer networking. If exist or possible, is layer notion still applied?
Muhammad Ikhwan Perwira's user avatar
-1 votes
1 answer
721 views

How Many Hidden Units in an LSTM? [duplicate]

Is there any rule of thumb for choosing the number of hidden units in an LSTM? Is it similar to hidden neurons in a regular feedforward neural network? I'm getting better results with my LSTM when I ...
Chris T's user avatar
  • 11
2 votes
0 answers
973 views

How many layers do GPT-3, AlphaFold 2, and DALL-E 2 have?

Unsuccessfully, I tried to find out the "depth" (definition below) in large neural networks such as GPT-3, AlphaFold 2, and DALL-E 2. Formally, my question is about their computational graph:...
keyboardAnt's user avatar
1 vote
0 answers
207 views

How to define a custom layer in Pytorch [closed]

I am new to PyTorch and seeking your help regarding a problem I have. I need to add a costume layer to a NN in training phase. Please see the figure which shows a simple DNN with the custom layer. NN ...
zstr's user avatar
  • 11
0 votes
2 answers
155 views

How do multimodal models establish connections between different modes?

I am specifically interested in data2vec, Meta's new model that can convert image, text, and sound data into a unified neural network representation. To my understanding, they did this through self-...
user3576467's user avatar
2 votes
0 answers
536 views

How does back propagation adjust the hidden layers' weights and biases?

I'm new to neural networks and trying to figure out its fundamentals but I cannot fully understand the back propagation algorithm. In back propagation, I understand we want to go backwards from the ...
KiarashRzg's user avatar
0 votes
2 answers
725 views

Do larger numbers of hidden layers have a bigger effect on a classification model's accuracy?

I trained different classification models using Keras with different numbers of hidden layers and the same number of neurons in each layer. What I found was the accuracy of the models decreased as the ...
Shonix3373's user avatar
20 votes
5 answers
3k views

Why does Batch Normalization work?

Adding BatchNorm layers improves training time and makes the whole deep model more stable. That's an experimental fact that is widely used in machine learning practice. My question is - why does it ...
Kostya's user avatar
  • 2,426
2 votes
1 answer
870 views

What's the purpose of layers without biases?

I noticed that the TensorFlow library includes a use_bias parameter for the Dense layer, which is set to ...
mark mark's user avatar
  • 763
0 votes
1 answer
75 views

If neurons performed the operation of an entire layer, would that make the neural network more effective?

(I have a very primitive understanding of neural networks, so please forgive the lack of technicality here.) I am used to seeing a neuron in a neural network as something that- Takes the inputs and ...
Bipasha's user avatar
  • 111
3 votes
1 answer
69 views

Are there deep neural networks that have inputs connected with deeper hidden layers?

Are there any architectures of deep neural networks that connect input neurons not only with the first hidden layer but also with deeper ones (red lines on the picture)? If so could you give some ...
GKozinski's user avatar
  • 1,240
2 votes
1 answer
693 views

Can the hidden layer prior to the ouput layer have less hidden units than the output layer?

I attended an introductory class about neural network and I had a question regarding how to choose the number of hidden units per hidden layer. I remember that the Professor saying that there is no ...
JoJolyne's user avatar
1 vote
1 answer
167 views

When are multiple hidden layers necessary?

I know that my question probably seems like being asked many times, but Ill try to be more speciffic: Limitations to my question: I am NOT asking about convolutional neural networks, so please, try ...
Igor's user avatar
  • 181
1 vote
1 answer
791 views

Why does the output shape of a Dense layer contain a batch size?

I understand that the batch size is the number of examples you pass into the neural network (NN). If the batch size is 10, it means you feed the NN 10 examples at once. Assuming I have an NN with a ...
jaksnak's user avatar
  • 13
1 vote
1 answer
2k views

How to determine the number of hidden layers and units of a deep auto-encoder?

I am using a deep autoencoder for my problem. However, the way I choose the number of hidden layers and hidden units in a hidden layer is still based on my feeling. The size of the model that ...
Truong Hoang's user avatar
0 votes
1 answer
926 views

Number of LSTM layers needed to learn a certain number of sequences

Theoretically, number of units for a LSTM layer is the number of hidden states or the max length of sequences as per my practice. For example, in Keras: ...
Dan D.'s user avatar
  • 1,283
1 vote
1 answer
34 views

How to work on different models for a given problem?

I am working on the MNIST data on my own. The idea is to use different values for the number of hidden layers, number of nodes in a given layer, etc. How do you organize these things while you are ...
Amar Parajuli's user avatar
0 votes
2 answers
278 views

How are non-linear surfaces formed in the training of a neural network? [closed]

Desperate trying to understand something for couple of weeks. All those questions are actually one big question.Please help me. Time-codes and screens in my question refer to this great(IMHO) 3d ...
Igor's user avatar
  • 181
4 votes
1 answer
111 views

Is it expected that adding an additional hidden layer to my 3-layer ANN reduces accuracy significantly?

I've been using several resources to implement my own artificial neural network package in C++. Among some of the resources I've been using are https://www.anotsorandomwalk.com/backpropagation-...
Chris's user avatar
  • 25
7 votes
1 answer
3k views

Do all neurons in a layer have the same activation function?

I'm new to machine learning (so excuse my nomenclature), and not being a python developer, I decided to jump in at the deep (no pun intended) end writing my own framework in C++. In my current design, ...
lfgtm's user avatar
  • 220
7 votes
3 answers
4k views

Does each filter in each convolution layer create a new image? [duplicate]

Say I have a CNN with this structure: input = 1 image (say, 30x30 RGB pixels) first convolution layer = 10 5x5 convolution filters second convolution layer = 5 3x3 convolution filters one dense layer ...
RocketNuts's user avatar
4 votes
2 answers
178 views

What could be the problem when a neural network with four hidden layers with the sigmoid activation function is not learning?

I have a large set of data points describing mappings of binary vectors to real-valued outputs. I am using TensorFlow, and would like to train a model to predict these relationships. I used four ...
Aggraj Gupta's user avatar
13 votes
4 answers
26k views

What exactly is a hidden state in an LSTM and RNN?

I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of ...
user8714896's user avatar
6 votes
1 answer
2k views

Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct?

I have an idea to find the optimal number of hidden neurons required in a neural network, but I'm not sure how accurate it is. Assuming that it has only 1 hidden layer, it is a classification problem ...
w13rfed's user avatar
  • 205
1 vote
2 answers
2k views

Why is there a sigmoid function in the hidden layer of a neural network? [duplicate]

I got this slide from CMU's lecture notes. The $x_i$s on the right are inputs and the $w_i$s are weights that get multiplied together then summed up at each hidden layer node. So I'm assuming this is ...
Gooby's user avatar
  • 351
1 vote
2 answers
400 views

How do we choose the activation function for each hidden node? [duplicate]

I am new to neural networks. I would like to use them as a fitting or forecasting method. A simple NN model that does not contain hidden layers, that is, the input nodes are directly connected to the ...
Nizar's user avatar
  • 111
1 vote
2 answers
364 views

Why is embedding important in NLP, and how does autoencoder work?

People say embedding is necessary in NLP because if using just the word indices, the efficiency is not high as similar words are supposed to be related to each other. However, I still don't truly get ...
Dan D.'s user avatar
  • 1,283
0 votes
1 answer
127 views

How many hidden layers are needed for this training data set

I'm trying to separate classes in 3D space, the data are as in the sketch below: There are 3 classes: 0,1,2; and with the look into the sketch, it seems that I need 3 planes to separate the classes, ...
Dan D.'s user avatar
  • 1,283
1 vote
1 answer
86 views

How does a single neuron in hidden layer affect training accuracy [duplicate]

I'm currently a student learning about AI Networks. I've came across a statement in one of my Professor's books that a FFBP (Feed-Forward Back-Propagation) Neural Network with a single hidden layer ...
Konrad Ł's user avatar
2 votes
1 answer
120 views

Does the encoding of a restricted Boltzmann machine improve with more layers?

I'm using a restricted Boltzmann machine (RBM) as an autoencoder. For now, I use a simple architecture of two layers, the input (~100 nodes) and the output (3 nodes) layers. I'm thinking to add more ...
Simone Morettini's user avatar
2 votes
0 answers
52 views

Multi-field text input for LSTM

I'm using LSTM to categorize medium-sized pieces of text. Each item to be categorized has several free-form text fields, in addition to several categorical fields. What is the best approach to using ...
Derek Hans's user avatar
1 vote
2 answers
8k views

How to chose dense layer size?

I am fine-tuning a VGG16 model on 20 classes with 500k images I was wondering how do you chose the size of the dense layer (the one before the prediction layer which has a size 20). I would prefer not ...
Hadrien Berthier's user avatar
2 votes
1 answer
531 views

Can particle swarm optimization be used to train neural networks with more than one hidden layer?

I've been thinking about the idea of replacing the classic gradient descent algorithm with an algorithm that is less sensitive to a local optimum. I was thinking about particle swarm optimization (PSO)...
gerard's user avatar
  • 73
2 votes
1 answer
147 views

Why are the nodes (or neurons) in neural networks depicted as circles?

Why are the nodes (or neurons) in neural networks depicted as circles? What is the difference between a circle and a box in diagrams of neural networks?
shrivathsa's user avatar
7 votes
3 answers
387 views

To what does the number of hidden layers in a neural network correspond?

In a neural network, the number of neurons in the hidden layer corresponds to the complexity of the model generated to map the inputs to output(s). More neurons creates a more complex function (and ...
SeeDerekEngineer's user avatar
3 votes
1 answer
2k views

How do I choose the size of the hidden state of a GRU?

I'm trying to understand how the size of the hidden state affects the GRU. For example, suppose I want to make a GRU count. I'm gonna feed it with three numbers, and I expect it to predict the ...
razvanc92's user avatar
  • 1,118
4 votes
1 answer
223 views

How does a single hidden layer affect output? [duplicate]

I'm learning about multilayer perceptrons, and I have a quick theory question in regards to hidden layer neurons. I know we can use two hidden layers to solve a non-linearly separable problem by ...
Howard P's user avatar
  • 153
1 vote
1 answer
769 views

What is the minimum number of neurons and hidden layers needed to learn a Boolean function that maps $N$ bits to $1$ bit?

Suppose I have a Boolean function that maps $N$ bits to $1$ bit. If I understand correctly, this function will have $2^{2^N}$ possible configurations of its truth table. What is the minimum number of ...
wil3's user avatar
  • 119
9 votes
1 answer
916 views

Why aren't there neural networks that connect the output of each layer to all next layers?

Why aren't there neural networks that connect the output of each layer to all next layers? For example, the output of layer 1 would be fed to the input of layers 2, 3, 4, etc. Beyond computational ...
Christopher Jernigan's user avatar
8 votes
2 answers
7k views

Why should the number of neurons in a hidden layer be a power of 2?

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. Is ...
dsfx3d's user avatar
  • 205
2 votes
0 answers
167 views

Data prepared to linear regression. Can I use it with backpropagation?

I'm studying a Master's Degree in Artificial Intelligence and I need to learn how to use the Java Neural Network Simulator, JavaNNS, program. In one practice I have to build a neural network to use ...
VansFannel's user avatar
5 votes
2 answers
2k views

What type of neural network would be most feasible for playing a realtime game?

For implementing a neural network algorithm that can play air hockey, I had two ideas for input, and I'm trying to figure out which design would be most viable. The output must be two analog values ...
Patrick Roberts's user avatar
3 votes
3 answers
119 views

Are these statements about the performance of neural networks as a function of the number of hidden layers contradictory?

In this note Justin Domke says that In practice, neural networks seem to usually find a reasonable solution when the number of layers is not too large, but find poor solutions when using more than, ...
Ming's user avatar
  • 41
1 vote
1 answer
120 views

How many layers and neurons on the layers does a neural network need for for sales prediction?

The inputs (features) and expected output for my ANN are these: Input 1: Product id (number, cast to double) Input 2: Year in the past (1900..2017, cast to double) Input 3: Month of year (1..12, cast ...
Dan D.'s user avatar
  • 1,283
6 votes
1 answer
361 views

How many nodes/hidden layers are required to solve a classification problem where the boundary is a sinusoidal function?

A single neuron is capable of forming a decision boundary between linearly seperable data. Is there any intuition as to how many, and in what configuration, would be necessary to correctly approximate ...
Ian Larson's user avatar
4 votes
1 answer
1k views

How can I understand this statement about RNNs and hidden layers?

In the lecture, there was a statement: Recurrent neural networks with multiple hidden layers are just a special case that has some of the hidden to hidden connections missing. I understand ...
Abhishek Bhatia's user avatar