Questions tagged [feedforward-neural-networks]
For questions related to feedforward neural networks (FFNNs), which are also sometimes called multilayer perceptrons, but these two expressions may not always be interchangeable.
96
questions
0
votes
1
answer
34
views
Loss Function not Decreasing
To practice what I learned about PyTorch, I gave myself the following problem:
Create a model that given a vector, predicts what the 2nd largest number in it is.
For example, ...
- 101
0
votes
0
answers
26
views
Neural Network with Incorrect Calculation Better than Correct One
I have designed my own neural network and discovered an error. During backpropagation, instead of inserting the Z-values into the derivative of the activation function, I inserted the A-values. The ...
- 1
0
votes
0
answers
28
views
What's the overall algorithm for population evolution in the NEAT algorithm?
I am implementing NEAT from scratch using Ruby, and I'm having a hard time understanding the necessary steps and overall algorithm of what happens between generations.
I have the ...
- 101
0
votes
1
answer
100
views
What does "position-wise" fully connected mean?
I understand the architecture of a position-wise feed-forward network as described in (https://nn.labml.ai/transformers/feed_forward.html). However I do not understand what "position wise" ...
0
votes
1
answer
47
views
How to optimise a FNN/MLP network with MSE (positive only loss) in C
I can create a FNN/MLP network in C but only g-p loss works, where g = ground truth and p = predicted.
What I don't understand is how MSE a positive only loss value can train a back propagation ...
user81637
1
vote
2
answers
114
views
Books that contains exclusively math problems/assignments in Deep Learning & Neural Networks
I am doing a Deep Learning Course.Suggest some books that contains exclusively math problems/assignments in Deep Learning & Neural Networks. I can understand that majority of the replies suggest &...
- 13
2
votes
1
answer
136
views
Neural Networks are universal approximators? - Exercice 20.1 UML
I'm working on this question which can be found at page 282 of "Understanding Machine Learning: From Theory to Algorithms" by Shai Shalev-Shwartz and Shai Ben-David.
The statement is as ...
- 21
0
votes
1
answer
46
views
chained linear regression models vs feed forward NN
I am trying to understand the difference between feedforward NN and chained linear regression models, if and why they can model nonlinear functions.
both are able to model non-linear dependencies
...
1
vote
1
answer
57
views
A Feedforward Neural Network (FNN) implemented with RMSProp optimization is exhibiting a tendency to overclassify instances into one particular class
I'm coding an FNN in Rust using the nalgebra crate. I coded the backpropagation based on this article from Brilliant (the link directly highlights the formulas' section I).
The issue
My network tends ...
- 13
0
votes
2
answers
65
views
How are hidden layers counted / semantically defined?
I'm working my way through how LLMs work and I understand how things work but it's not clear to me exactly what is semantically defined as a "layer".
Using the following FFN as an example:
...
- 213
0
votes
0
answers
80
views
Why does my loss function fluctuate so much?
I have a loss function that I'm trying to maximise using a neural network.
While it does appear to increase and plateau over the training, it does so in a very "noisy" manner, spiking up and ...
- 83
1
vote
0
answers
40
views
Is there a neural network architecture specialized for mapping lower-to-higher-dimensional data?
I am building a neural network that takes in a set of 86 parameters (primarily architecture-related, such as building floor area, kitchen size, number of a certain type of furniture, etc.) and outputs ...
- 143
1
vote
2
answers
166
views
How does a multidimensional vector get fed into a single node in a neural network?
I mostly develop neural networks completely from scratch, like without libraries. I've been seeing, especially in NLP tasks, entire vectors, often representing words, get fed into a single node. I'm ...
- 11
0
votes
1
answer
57
views
Shape of biases in Transformer's Feedforward Network
In transformer network (Vaswani et al., 2017), the feedforward networks have equation:
$$\mathrm{FNN}(x) = \max(0, xW_1 + b_1) W_2 + b_2$$
where $x \in \mathbb{R}^{n \times d_\mathrm{model}}$, $W_1 \...
- 23
1
vote
1
answer
261
views
How can an MLP be implemented with convolutional layers?
I am studying the architecture of the network pointnet, specifically the MLPs stages of the pipeline highlighted in red in the following image (taken from the author page here):
It is strange to find ...
12
votes
5
answers
3k
views
Why is a bias parameter needed in neural networks?
I have read several resources, including previously asked questions such as this. I have also read arguments related to intercepts needed to separate linearly separable data. If my neural network can ...
- 326
1
vote
0
answers
69
views
How to code mathematical functions in terms of neural networks?
I'm exploring how to solve differential equations using neural networks and discovered this Lagaris et. al. paper. In the paper, the solution to a differential equation is written as a mathematical ...
- 11
1
vote
1
answer
79
views
Find maxima and minima of feed forward neural network given interval inputs [closed]
I have a feed forward neural network defined in pytorch as follows:
...
- 13
0
votes
2
answers
1k
views
Are neural networks a strict special case of a transformer?
Since transformers contain a neural network, are they a strict generalisation of standard feedforward neural networks? In what ways can transformers be interpreted as a generalisation and abstraction ...
- 131
1
vote
1
answer
104
views
What type of neural network has an unorganized structure?
I am looking for a network that has an unorganized structure like this, is feed-forward, does not have back-propagation functionality, and is trained with a genetic algorithm.
What would I be looking ...
- 17
1
vote
2
answers
64
views
What are all the possible usages of 'multilayer perceptron'?
The term 'multilayer perceptron' has been used in literature in various ways in the literature.
I am presenting some of them below
As a feed-forward neural network [1].
As a fully connected feed-...
- 3,960
0
votes
3
answers
343
views
How to deal with an unbalanced dataset?
I'm constructing a feed forward neural network that predicts whether a patient will get a stroke or not. However, my dataset is very unbalanced. Out of 5111 rows, 250 contain patients that have had a ...
- 9
1
vote
1
answer
685
views
Is "node embedding" in GNN analogous to "hidden layer" of FFN?
So in Graph Neural Network (GNN) we have node embeddings which is a feature vector that describes the node, is it analogous to hidden layer of Artificial neural network such as feed-forward neural ...
- 145
2
votes
2
answers
959
views
How can the input order of pairs into a neural network not matter (i.e. symmetry)?
Let me explain, suppose we are building a neural network that predicts if two items are similar or not. This is a classification task with hard labels (0, 1) of examples of similar and dissimilar ...
- 159
0
votes
1
answer
132
views
How many layers and neurons in a FFNN do I need to make it equivalent to a CNN?
I started to learn machine learning early, and I studied the convolutional neural network and its ability to understand images and how it helps to reduce the number of parameters that need to be tuned....
1
vote
0
answers
54
views
Can I help my neural network if I know the sign of the relationships between inputs and outputs
I am attempting to train a neural network where I can say the following:
For most inputs, I know the sign of the relationship between that input and several specific outputs. I.e. whatever set of ...
- 111
0
votes
1
answer
216
views
How to uniquely associate a directed graph with a feedforward neural network?
I want to write an algorithm that returns a unique directed graph (an adjacency matrix) that represents the structure of a given feedforward neural network (FNN). My idea is to deconstruct the FNN ...
- 23
2
votes
0
answers
138
views
Does there exist functions for which the necessary number of nodes in a shallow neural network tends to infinity as approximation error tends to 0?
The Universal Approximation Theorem states (roughly) that any continuous function can be approximated to within an arbitrary precision $\varepsilon>0$ by a feedforward neural network with one ...
- 23
2
votes
1
answer
170
views
Closed networks vs Networks with a removed delay to predict new data
I've come across two types of neural networks to predict, both from Matlab, the closed structure and the net that removes one delay to find new data.
From Matlab's app generated scripts we see:
% ...
1
vote
0
answers
60
views
ReLU function converging to local optimum in one case and diverging in the other one
I implemented a simple neural network with 1 hidden layer. I used ReLU as activation function for the hidden layer and the output layer just uses the linear function.
To check my implementation I ...
- 147
0
votes
1
answer
114
views
About the choice of the activation functions in the Multilayer Perceptron, and on what does this depends?
I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are ...
2
votes
1
answer
434
views
Why is the backpropagation algorithm used to train the multilayer perceptron?
I've read in the book Neural Network Design, by Martin Hagan et al. (chapter 11), that, to train the feed-forward neural network (aka multilayer perceptron), one uses the backpropagation algorithm.
...
1
vote
1
answer
123
views
Issue with graphical interpretation of the universal approximation theorem
This article attempts to provide a graphical justification of the universal approximation theorem.
It succeeds in showing that a linear combination of two sigmoids can produce essentially a bounded ...
- 39
1
vote
1
answer
88
views
Can RNNs get inputs and produce outputs similar to the inputs and outputs of FFNNs?
RNN and LSTM models have many architectures that can be modified. We can also compose their input and output data. However, in the examples that I found on the web, the inputs and outputs of RNNs/...
- 45
1
vote
1
answer
616
views
What are examples of good free books that cover the back-propagation algorithm?
What are examples of good free books that cover the back-propagation used to train multilayer perceptrons? I've just started to learn about artificial neural networks, so I'm looking for books that ...
2
votes
0
answers
255
views
Bias gradient of layer before batch normalization always zero
From the original paper and this post we have that batch normalization backpropagation can be formulated as
I'm interested in the derivative of the previous layer outputs $x_i=\sigma(w X_i+b)$ with ...
- 21
6
votes
1
answer
1k
views
Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?
Why does a vanilla feedforward neural network only accept a fixed input size, while RNNs are capable of taking a series of inputs with no predetermined limit on the size? Can anyone elaborate on this ...
- 63
5
votes
2
answers
293
views
Is it really possible to create the "Perfect Cylinder" used in Universal Approximation Theorem for 1-hidden layer Neural Network?
There are proofs for the universal approximation theorem with just 1 hidden layer.
The proof goes like this:
Create a "bump" function using 2 neurons.
Create (infinitely) many of these ...
- 133
6
votes
1
answer
2k
views
How to express a fully connected neural network succintly using linear algebra?
I'm currently reading the paper Federated Learning with Matched Averaging (2020), where the authors claim:
A basic fully connected (FC) NN can be formulated as: $\hat{y} = \sigma(xW_1)W_2$ [...]
...
- 83
1
vote
1
answer
3k
views
What is the space complexity for training a neural network using back-propagation?
Suppose that a simple feedforward neural network (FFNN) contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_l$ nodes in each layer. What is the space complexity to train this FFNN ...
- 111
2
votes
0
answers
328
views
Why should variance(output) equal variance(input) in Xavier Initialisation?
In a lot of explanations online for Xavier Initialization, I see the following:
With each passing layer, we want the variance to remain the same. This helps us keep the signal from exploding to a ...
- 177
2
votes
2
answers
301
views
How to tell a neural network that: "your i-th input is special"
Assume that I have a fully connected network that takes in a vector containing 1025 elements. First 1024 elements are related to ...
- 420
1
vote
2
answers
1k
views
How is the error calculated with multiple output neurons in the neural network?
Machine Learning books generally explains that the error calculated for a given sample $i$ is:
$e_i = y_i - \hat{y_i}$
Where $\hat{y}$ is the target output and $y$ is the actual output given by the ...
- 111
2
votes
1
answer
841
views
Can the hidden layer prior to the ouput layer have less hidden units than the output layer?
I attended an introductory class about neural network and I had a question regarding how to choose the number of hidden units per hidden layer.
I remember that the Professor saying that there is no ...
- 35
3
votes
0
answers
646
views
How to use residual learning applied to fully connected networks?
Is there any reason why skip connections would not provide the same benefits to fully connected layers as it does for convolutional?
I've read the ResNet paper and it says that the applications should ...
- 143
1
vote
0
answers
39
views
What's the difference between RNNs and Feed Forward Neural Networks if a fixed size vector can preserve sequential information?
I was watching a Youtube video in which the problem of trying to predict the last word in a sentence was posed. The sentence was "I took my cat for a" and the last word was "walk"....
1
vote
1
answer
75
views
What's a good neural network for this problem?
I am very new to the field of AI so please bear with me.
Say there is a dice with three sides, -1,0 and 1, and I want to predict which side it lands on (so only one output is needed I guess). The ...
- 113
2
votes
1
answer
159
views
What is the Preferred Mathematical Representation for a Forward Pass in a Neural Network?
I know this may be a question of semantics but I always see different articles explain forward pass slightly different. e.g. Sometimes they represent a forward pass to a hidden layer in a standard ...
- 121
0
votes
3
answers
1k
views
Why isn't the loss of my neural network reduced after 2500 iterations?
I have developed a basic feedforward neural network from scratch to classify whether image is of cat or not cat. It works fine, but after 2500 iterations, my cost function is not reducing properly.
...
- 9
1
vote
2
answers
121
views
Can we achieve what a CNN can do with just a normal neural network?
When I was learning about neural networks, I saw that a complex neural network can understand the MNIST dataset and a simple convolution network can also understand the same. So I would like to know ...
- 111