Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [feedforward-neural-networks]

For questions related to feedforward neural networks (FFNNs), which are also sometimes called multilayer perceptrons, but these two expressions may not always be interchangeable.

Filter by
Sorted by
Tagged with
0 votes
0 answers
24 views

Neural Network with Incorrect Calculation Better than Correct One

I have designed my own neural network and discovered an error. During backpropagation, instead of inserting the Z-values into the derivative of the activation function, I inserted the A-values. The ...
Apro9991's user avatar
0 votes
0 answers
22 views

What's the overall algorithm for population evolution in the NEAT algorithm?

I am implementing NEAT from scratch using Ruby, and I'm having a hard time understanding the necessary steps and overall algorithm of what happens between generations. I have the ...
Thiago Belem's user avatar
0 votes
1 answer
30 views

What does "position-wise" fully connected mean?

I understand the architecture of a position-wise feed-forward network as described in (https://nn.labml.ai/transformers/feed_forward.html). However I do not understand what "position wise" ...
Homer Sanchez's user avatar
0 votes
1 answer
42 views

How to optimise a FNN/MLP network with MSE (positive only loss) in C

I can create a FNN/MLP network in C but only g-p loss works, where g = ground truth and p = predicted. What I don't understand is how MSE a positive only loss value can train a back propagation ...
user avatar
1 vote
2 answers
76 views

Books that contains exclusively math problems/assignments in Deep Learning & Neural Networks

I am doing a Deep Learning Course.Suggest some books that contains exclusively math problems/assignments in Deep Learning & Neural Networks. I can understand that majority of the replies suggest &...
GKK's user avatar
  • 13
2 votes
1 answer
123 views

Neural Networks are universal approximators? - Exercice 20.1 UML

I'm working on this question which can be found at page 282 of "Understanding Machine Learning: From Theory to Algorithms" by Shai Shalev-Shwartz and Shai Ben-David. The statement is as ...
mabed's user avatar
  • 21
0 votes
1 answer
40 views

chained linear regression models vs feed forward NN

I am trying to understand the difference between feedforward NN and chained linear regression models, if and why they can model nonlinear functions. both are able to model non-linear dependencies ...
Klembajnsztajn's user avatar
1 vote
1 answer
54 views

A Feedforward Neural Network (FNN) implemented with RMSProp optimization is exhibiting a tendency to overclassify instances into one particular class

I'm coding an FNN in Rust using the nalgebra crate. I coded the backpropagation based on this article from Brilliant (the link directly highlights the formulas' section I). The issue My network tends ...
Evry's user avatar
  • 13
0 votes
2 answers
64 views

How are hidden layers counted / semantically defined?

I'm working my way through how LLMs work and I understand how things work but it's not clear to me exactly what is semantically defined as a "layer". Using the following FFN as an example: ...
Grant Curell's user avatar
0 votes
0 answers
56 views

Why does my loss function fluctuate so much?

I have a loss function that I'm trying to maximise using a neural network. While it does appear to increase and plateau over the training, it does so in a very "noisy" manner, spiking up and ...
VJ123's user avatar
  • 73
1 vote
0 answers
39 views

Is there a neural network architecture specialized for mapping lower-to-higher-dimensional data?

I am building a neural network that takes in a set of 86 parameters (primarily architecture-related, such as building floor area, kitchen size, number of a certain type of furniture, etc.) and outputs ...
JS4137's user avatar
  • 143
1 vote
1 answer
98 views

How does a multidimensional vector get fed into a single node in a neural network?

I mostly develop neural networks completely from scratch, like without libraries. I've been seeing, especially in NLP tasks, entire vectors, often representing words, get fed into a single node. I'm ...
Jake StBu's user avatar
0 votes
0 answers
12 views

How are groups created in maxout units when dividing the set of inputs 𝑧 into groups of 𝑘 values?

I don't get $G^(i)$the set of indices into the inputs for group $i$, $\{(i −1)k+ 1, . . . , ik\}$ when creating a maxout units/function, these thing that outputs the maximum element of groups: $$g(z)...
Revolucion for Monica's user avatar
0 votes
1 answer
44 views

Shape of biases in Transformer's Feedforward Network

In transformer network (Vaswani et al., 2017), the feedforward networks have equation: $$\mathrm{FNN}(x) = \max(0, xW_1 + b_1) W_2 + b_2$$ where $x \in \mathbb{R}^{n \times d_\mathrm{model}}$, $W_1 \...
poglhar's user avatar
  • 23
1 vote
0 answers
177 views

How can an MLP be implemented with convolutional layers?

I am studying the architecture of the network pointnet, specifically the MLPs stages of the pipeline highlighted in red in the following image (taken from the author page here): It is strange to find ...
Jacob Morales Gonzalez's user avatar
12 votes
5 answers
3k views

Why is a bias parameter needed in neural networks?

I have read several resources, including previously asked questions such as this. I have also read arguments related to intercepts needed to separate linearly separable data. If my neural network can ...
SajanGohil's user avatar
1 vote
0 answers
60 views

How to code mathematical functions in terms of neural networks?

I'm exploring how to solve differential equations using neural networks and discovered this Lagaris et. al. paper. In the paper, the solution to a differential equation is written as a mathematical ...
mombash's user avatar
  • 11
1 vote
1 answer
75 views

Find maxima and minima of feed forward neural network given interval inputs [closed]

I have a feed forward neural network defined in pytorch as follows: ...
peocje's user avatar
  • 13
0 votes
2 answers
1k views

Are neural networks a strict special case of a transformer?

Since transformers contain a neural network, are they a strict generalisation of standard feedforward neural networks? In what ways can transformers be interpreted as a generalisation and abstraction ...
Featherball's user avatar
1 vote
1 answer
103 views

What type of neural network has an unorganized structure?

I am looking for a network that has an unorganized structure like this, is feed-forward, does not have back-propagation functionality, and is trained with a genetic algorithm. What would I be looking ...
coder's user avatar
  • 17
1 vote
2 answers
62 views

What are all the possible usages of 'multilayer perceptron'?

The term 'multilayer perceptron' has been used in literature in various ways in the literature. I am presenting some of them below As a feed-forward neural network [1]. As a fully connected feed-...
hanugm's user avatar
  • 3,890
0 votes
3 answers
282 views

How to deal with an unbalanced dataset?

I'm constructing a feed forward neural network that predicts whether a patient will get a stroke or not. However, my dataset is very unbalanced. Out of 5111 rows, 250 contain patients that have had a ...
JanHudec's user avatar
1 vote
1 answer
644 views

Is "node embedding" in GNN analogous to "hidden layer" of FFN?

So in Graph Neural Network (GNN) we have node embeddings which is a feature vector that describes the node, is it analogous to hidden layer of Artificial neural network such as feed-forward neural ...
user0193's user avatar
  • 145
2 votes
2 answers
883 views

How can the input order of pairs into a neural network not matter (i.e. symmetry)?

Let me explain, suppose we are building a neural network that predicts if two items are similar or not. This is a classification task with hard labels (0, 1) of examples of similar and dissimilar ...
Michael's user avatar
  • 159
0 votes
1 answer
131 views

How many layers and neurons in a FFNN do I need to make it equivalent to a CNN?

I started to learn machine learning early, and I studied the convolutional neural network and its ability to understand images and how it helps to reduce the number of parameters that need to be tuned....
Mahmoud Kanbar's user avatar
1 vote
0 answers
50 views

Can I help my neural network if I know the sign of the relationships between inputs and outputs

I am attempting to train a neural network where I can say the following: For most inputs, I know the sign of the relationship between that input and several specific outputs. I.e. whatever set of ...
Mick's user avatar
  • 111
0 votes
1 answer
197 views

How to uniquely associate a directed graph with a feedforward neural network?

I want to write an algorithm that returns a unique directed graph (an adjacency matrix) that represents the structure of a given feedforward neural network (FNN). My idea is to deconstruct the FNN ...
GraftCraft's user avatar
1 vote
0 answers
133 views

Does there exist functions for which the necessary number of nodes in a shallow neural network tends to infinity as approximation error tends to 0?

The Universal Approximation Theorem states (roughly) that any continuous function can be approximated to within an arbitrary precision $\varepsilon>0$ by a feedforward neural network with one ...
GraftCraft's user avatar
2 votes
1 answer
165 views

Closed networks vs Networks with a removed delay to predict new data

I've come across two types of neural networks to predict, both from Matlab, the closed structure and the net that removes one delay to find new data. From Matlab's app generated scripts we see: % ...
Verónica Rmz.'s user avatar
1 vote
0 answers
57 views

ReLU function converging to local optimum in one case and diverging in the other one

I implemented a simple neural network with 1 hidden layer. I used ReLU as activation function for the hidden layer and the output layer just uses the linear function. To check my implementation I ...
SAGALPREET SINGH's user avatar
0 votes
1 answer
112 views

About the choice of the activation functions in the Multilayer Perceptron, and on what does this depends?

I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are ...
Verónica Rmz.'s user avatar
2 votes
1 answer
424 views

Why is the backpropagation algorithm used to train the multilayer perceptron?

I've read in the book Neural Network Design, by Martin Hagan et al. (chapter 11), that, to train the feed-forward neural network (aka multilayer perceptron), one uses the backpropagation algorithm. ...
Verónica Rmz.'s user avatar
1 vote
1 answer
115 views

Issue with graphical interpretation of the universal approximation theorem

This article attempts to provide a graphical justification of the universal approximation theorem. It succeeds in showing that a linear combination of two sigmoids can produce essentially a bounded ...
sangstar's user avatar
1 vote
1 answer
84 views

Can RNNs get inputs and produce outputs similar to the inputs and outputs of FFNNs?

RNN and LSTM models have many architectures that can be modified. We can also compose their input and output data. However, in the examples that I found on the web, the inputs and outputs of RNNs/...
Green's user avatar
  • 45
1 vote
1 answer
549 views

What are examples of good free books that cover the back-propagation algorithm?

What are examples of good free books that cover the back-propagation used to train multilayer perceptrons? I've just started to learn about artificial neural networks, so I'm looking for books that ...
Verónica Rmz.'s user avatar
2 votes
0 answers
231 views

Bias gradient of layer before batch normalization always zero

From the original paper and this post we have that batch normalization backpropagation can be formulated as I'm interested in the derivative of the previous layer outputs $x_i=\sigma(w X_i+b)$ with ...
Matthias's user avatar
6 votes
1 answer
1k views

Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?

Why does a vanilla feedforward neural network only accept a fixed input size, while RNNs are capable of taking a series of inputs with no predetermined limit on the size? Can anyone elaborate on this ...
Daniel's user avatar
  • 63
5 votes
2 answers
282 views

Is it really possible to create the "Perfect Cylinder" used in Universal Approximation Theorem for 1-hidden layer Neural Network?

There are proofs for the universal approximation theorem with just 1 hidden layer. The proof goes like this: Create a "bump" function using 2 neurons. Create (infinitely) many of these ...
KoKlA's user avatar
  • 133
6 votes
1 answer
2k views

How to express a fully connected neural network succintly using linear algebra?

I'm currently reading the paper Federated Learning with Matched Averaging (2020), where the authors claim: A basic fully connected (FC) NN can be formulated as: $\hat{y} = \sigma(xW_1)W_2$ [...] ...
user1360448's user avatar
1 vote
1 answer
3k views

What is the space complexity for training a neural network using back-propagation?

Suppose that a simple feedforward neural network (FFNN) contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_l$ nodes in each layer. What is the space complexity to train this FFNN ...
Ritika Gupta's user avatar
2 votes
0 answers
302 views

Why should variance(output) equal variance(input) in Xavier Initialisation?

In a lot of explanations online for Xavier Initialization, I see the following: With each passing layer, we want the variance to remain the same. This helps us keep the signal from exploding to a ...
THAT_AI_GUY's user avatar
2 votes
2 answers
299 views

How to tell a neural network that: "your i-th input is special"

Assume that I have a fully connected network that takes in a vector containing 1025 elements. First 1024 elements are related to ...
amin's user avatar
  • 420
1 vote
2 answers
1k views

How is the error calculated with multiple output neurons in the neural network?

Machine Learning books generally explains that the error calculated for a given sample $i$ is: $e_i = y_i - \hat{y_i}$ Where $\hat{y}$ is the target output and $y$ is the actual output given by the ...
joann2555's user avatar
  • 111
2 votes
1 answer
780 views

Can the hidden layer prior to the ouput layer have less hidden units than the output layer?

I attended an introductory class about neural network and I had a question regarding how to choose the number of hidden units per hidden layer. I remember that the Professor saying that there is no ...
JoJolyne's user avatar
3 votes
0 answers
630 views

How to use residual learning applied to fully connected networks?

Is there any reason why skip connections would not provide the same benefits to fully connected layers as it does for convolutional? I've read the ResNet paper and it says that the applications should ...
rocksNwaves's user avatar
1 vote
0 answers
39 views

What's the difference between RNNs and Feed Forward Neural Networks if a fixed size vector can preserve sequential information?

I was watching a Youtube video in which the problem of trying to predict the last word in a sentence was posed. The sentence was "I took my cat for a" and the last word was "walk"....
Prabhav Arora's user avatar
1 vote
1 answer
74 views

What's a good neural network for this problem?

I am very new to the field of AI so please bear with me. Say there is a dice with three sides, -1,0 and 1, and I want to predict which side it lands on (so only one output is needed I guess). The ...
minusatwelfth's user avatar
2 votes
1 answer
151 views

What is the Preferred Mathematical Representation for a Forward Pass in a Neural Network?

I know this may be a question of semantics but I always see different articles explain forward pass slightly different. e.g. Sometimes they represent a forward pass to a hidden layer in a standard ...
Coldchain9's user avatar
0 votes
3 answers
922 views

Why isn't the loss of my neural network reduced after 2500 iterations?

I have developed a basic feedforward neural network from scratch to classify whether image is of cat or not cat. It works fine, but after 2500 iterations, my cost function is not reducing properly. ...
Siddarth's user avatar
1 vote
2 answers
110 views

Can we achieve what a CNN can do with just a normal neural network?

When I was learning about neural networks, I saw that a complex neural network can understand the MNIST dataset and a simple convolution network can also understand the same. So I would like to know ...
Shrijit Basak's user avatar