Questions tagged [feedforward-neural-networks]
For questions related to feedforward neural networks (FFNNs), which are also sometimes called multilayer perceptrons, but these two expressions may not always be interchangeable.
92
questions
1
vote
1
answer
38
views
A Feedforward Neural Network (FNN) implemented with RMSProp optimization is exhibiting a tendency to overclassify instances into one particular class
hope you're alright.
I'm coding an FNN in Rust using the nalgebra crate. I coded the backpropagation based on this article from Brilliant (the link directly highlights the formulas' section I).
The ...
- 13
0
votes
2
answers
50
views
How are hidden layers counted / semantically defined?
I'm working my way through how LLMs work and I understand how things work but it's not clear to me exactly what is semantically defined as a "layer".
Using the following FFN as an example:
...
- 101
0
votes
0
answers
44
views
Why does my loss function fluctuate so much?
I have a loss function that I'm trying to maximise using a neural network.
While it does appear to increase and plateau over the training, it does so in a very "noisy" manner, spiking up and ...
- 73
1
vote
0
answers
39
views
Is there a neural network architecture specialized for mapping lower-to-higher-dimensional data?
I am building a neural network that takes in a set of 86 parameters (primarily architecture-related, such as building floor area, kitchen size, number of a certain type of furniture, etc.) and outputs ...
- 143
1
vote
1
answer
66
views
How does a multidimensional vector get fed into a single node in a neural network?
I mostly develop neural networks completely from scratch, like without libraries. I've been seeing, especially in NLP tasks, entire vectors, often representing words, get fed into a single node. I'm ...
- 11
0
votes
0
answers
11
views
How are groups created in maxout units when dividing the set of inputs 𝑧 into groups of 𝑘 values?
I don't get $G^(i)$the set of indices into the inputs for group $i$, $\{(i −1)k+ 1, . . . , ik\}$ when creating a maxout units/function, these thing that outputs the maximum element of groups: $$g(z)...
0
votes
1
answer
42
views
Shape of biases in Transformer's Feedforward Network
In transformer network (Vaswani et al., 2017), the feedforward networks have equation:
$$\mathrm{FNN}(x) = \max(0, xW_1 + b_1) W_2 + b_2$$
where $x \in \mathbb{R}^{n \times d_\mathrm{model}}$, $W_1 \...
- 23
1
vote
0
answers
109
views
How can an MLP be implemented with convolutional layers?
I am studying the architecture of the network pointnet, specifically the MLPs stages of the pipeline highlighted in red in the following image (taken from the author page here):
It is strange to find ...
0
votes
0
answers
48
views
Backpropagation of position-wise feedforward neural network
I have read a paper entitled "Attention is all you need" by Vaswani et al. (2017). This paper use the so-called position-wise feedforward neural network, where the input of this network is a ...
- 23
0
votes
0
answers
9
views
Why is avoiding normalized models a practical solution for reducing the complexity in NNLM?
In the paper Efficient Estimation of Word Representations in Vector Space, the authors say that "avoiding normalized models completely by using models that are not normalized during training"...
- 1
12
votes
5
answers
2k
views
Why is a bias parameter needed in neural networks?
I have read several resources, including previously asked questions such as this. I have also read arguments related to intercepts needed to separate linearly separable data. If my neural network can ...
- 326
0
votes
0
answers
86
views
When should I think of using the Forward Forward algorithm?
Recently I read the paper named "The Forward-Forward Algorithm: Some Preliminary Investigations" and I was wondering what cases I should think of using it on.
So according to the paper If we ...
1
vote
0
answers
57
views
How to code mathematical functions in terms of neural networks?
I'm exploring how to solve differential equations using neural networks and discovered this Lagaris et. al. paper. In the paper, the solution to a differential equation is written as a mathematical ...
- 11
1
vote
1
answer
67
views
Find maxima and minima of feed forward neural network given interval inputs [closed]
I have a feed forward neural network defined in pytorch as follows:
...
- 13
0
votes
2
answers
767
views
Are neural networks a strict special case of a transformer?
Since transformers contain a neural network, are they a strict generalisation of standard feedforward neural networks? In what ways can transformers be interpreted as a generalisation and abstraction ...
- 131
1
vote
1
answer
102
views
What type of neural network has an unorganized structure?
I am looking for a network that has an unorganized structure like this, is feed-forward, does not have back-propagation functionality, and is trained with a genetic algorithm.
What would I be looking ...
- 17
1
vote
2
answers
59
views
What are all the possible usages of 'multilayer perceptron'?
The term 'multilayer perceptron' has been used in literature in various ways in the literature.
I am presenting some of them below
As a feed-forward neural network [1].
As a fully connected feed-...
- 3,622
0
votes
3
answers
215
views
How to deal with an unbalanced dataset?
I'm constructing a feed forward neural network that predicts whether a patient will get a stroke or not. However, my dataset is very unbalanced. Out of 5111 rows, 250 contain patients that have had a ...
- 9
1
vote
1
answer
607
views
Is "node embedding" in GNN analogous to "hidden layer" of FFN?
So in Graph Neural Network (GNN) we have node embeddings which is a feature vector that describes the node, is it analogous to hidden layer of Artificial neural network such as feed-forward neural ...
- 145
2
votes
2
answers
776
views
How can the input order of pairs into a neural network not matter (i.e. symmetry)?
Let me explain, suppose we are building a neural network that predicts if two items are similar or not. This is a classification task with hard labels (0, 1) of examples of similar and dissimilar ...
- 159
0
votes
1
answer
120
views
How many layers and neurons in a FFNN do I need to make it equivalent to a CNN?
I started to learn machine learning early, and I studied the convolutional neural network and its ability to understand images and how it helps to reduce the number of parameters that need to be tuned....
1
vote
0
answers
47
views
Can I help my neural network if I know the sign of the relationships between inputs and outputs
I am attempting to train a neural network where I can say the following:
For most inputs, I know the sign of the relationship between that input and several specific outputs. I.e. whatever set of ...
- 111
0
votes
1
answer
181
views
How to uniquely associate a directed graph with a feedforward neural network?
I want to write an algorithm that returns a unique directed graph (an adjacency matrix) that represents the structure of a given feedforward neural network (FNN). My idea is to deconstruct the FNN ...
- 13
1
vote
0
answers
127
views
Does there exist functions for which the necessary number of nodes in a shallow neural network tends to infinity as approximation error tends to 0?
The Universal Approximation Theorem states (roughly) that any continuous function can be approximated to within an arbitrary precision $\varepsilon>0$ by a feedforward neural network with one ...
- 13
2
votes
1
answer
156
views
Closed networks vs Networks with a removed delay to predict new data
I've come across two types of neural networks to predict, both from Matlab, the closed structure and the net that removes one delay to find new data.
From Matlab's app generated scripts we see:
% ...
1
vote
0
answers
57
views
ReLU function converging to local optimum in one case and diverging in the other one
I implemented a simple neural network with 1 hidden layer. I used ReLU as activation function for the hidden layer and the output layer just uses the linear function.
To check my implementation I ...
- 107
0
votes
1
answer
105
views
About the choice of the activation functions in the Multilayer Perceptron, and on what does this depends?
I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are ...
2
votes
1
answer
405
views
Why is the backpropagation algorithm used to train the multilayer perceptron?
I've read in the book Neural Network Design, by Martin Hagan et al. (chapter 11), that, to train the feed-forward neural network (aka multilayer perceptron), one uses the backpropagation algorithm.
...
1
vote
1
answer
96
views
Issue with graphical interpretation of the universal approximation theorem
This article attempts to provide a graphical justification of the universal approximation theorem.
It succeeds in showing that a linear combination of two sigmoids can produce essentially a bounded ...
- 39
1
vote
1
answer
79
views
Can RNNs get inputs and produce outputs similar to the inputs and outputs of FFNNs?
RNN and LSTM models have many architectures that can be modified. We can also compose their input and output data. However, in the examples that I found on the web, the inputs and outputs of RNNs/...
- 45
1
vote
1
answer
457
views
What are examples of good free books that cover the back-propagation algorithm?
What are examples of good free books that cover the back-propagation used to train multilayer perceptrons? I've just started to learn about artificial neural networks, so I'm looking for books that ...
2
votes
0
answers
210
views
Bias gradient of layer before batch normalization always zero
From the original paper and this post we have that batch normalization backpropagation can be formulated as
I'm interested in the derivative of the previous layer outputs $x_i=\sigma(w X_i+b)$ with ...
- 21
6
votes
1
answer
1k
views
Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?
Why does a vanilla feedforward neural network only accept a fixed input size, while RNNs are capable of taking a series of inputs with no predetermined limit on the size? Can anyone elaborate on this ...
- 63
5
votes
2
answers
268
views
Is it really possible to create the "Perfect Cylinder" used in Universal Approximation Theorem for 1-hidden layer Neural Network?
There are proofs for the universal approximation theorem with just 1 hidden layer.
The proof goes like this:
Create a "bump" function using 2 neurons.
Create (infinitely) many of these ...
- 133
6
votes
1
answer
1k
views
How to express a fully connected neural network succintly using linear algebra?
I'm currently reading the paper Federated Learning with Matched Averaging (2020), where the authors claim:
A basic fully connected (FC) NN can be formulated as: $\hat{y} = \sigma(xW_1)W_2$ [...]
...
- 83
1
vote
1
answer
2k
views
What is the space complexity for training a neural network using back-propagation?
Suppose that a simple feedforward neural network (FFNN) contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_l$ nodes in each layer. What is the space complexity to train this FFNN ...
- 111
2
votes
0
answers
276
views
Why should variance(output) equal variance(input) in Xavier Initialisation?
In a lot of explanations online for Xavier Initialization, I see the following:
With each passing layer, we want the variance to remain the same. This helps us keep the signal from exploding to a ...
- 177
2
votes
2
answers
294
views
How to tell a neural network that: "your i-th input is special"
Assume that I have a fully connected network that takes in a vector containing 1025 elements. First 1024 elements are related to ...
- 420
1
vote
2
answers
1k
views
How is the error calculated with multiple output neurons in the neural network?
Machine Learning books generally explains that the error calculated for a given sample $i$ is:
$e_i = y_i - \hat{y_i}$
Where $\hat{y}$ is the target output and $y$ is the actual output given by the ...
- 111
2
votes
1
answer
693
views
Can the hidden layer prior to the ouput layer have less hidden units than the output layer?
I attended an introductory class about neural network and I had a question regarding how to choose the number of hidden units per hidden layer.
I remember that the Professor saying that there is no ...
- 35
3
votes
0
answers
586
views
How to use residual learning applied to fully connected networks?
Is there any reason why skip connections would not provide the same benefits to fully connected layers as it does for convolutional?
I've read the ResNet paper and it says that the applications should ...
- 143
1
vote
0
answers
38
views
What's the difference between RNNs and Feed Forward Neural Networks if a fixed size vector can preserve sequential information?
I was watching a Youtube video in which the problem of trying to predict the last word in a sentence was posed. The sentence was "I took my cat for a" and the last word was "walk"....
1
vote
1
answer
73
views
What's a good neural network for this problem?
I am very new to the field of AI so please bear with me.
Say there is a dice with three sides, -1,0 and 1, and I want to predict which side it lands on (so only one output is needed I guess). The ...
- 113
2
votes
1
answer
130
views
What is the Preferred Mathematical Representation for a Forward Pass in a Neural Network?
I know this may be a question of semantics but I always see different articles explain forward pass slightly different. e.g. Sometimes they represent a forward pass to a hidden layer in a standard ...
- 121
0
votes
2
answers
704
views
Why isn't the loss of my neural network reduced after 2500 iterations?
I have developed a basic feedforward neural network from scratch to classify whether image is of cat or not cat. It works fine, but after 2500 iterations, my cost function is not reducing properly.
...
- 1
1
vote
2
answers
105
views
Can we achieve what a CNN can do with just a normal neural network?
When I was learning about neural networks, I saw that a complex neural network can understand the MNIST dataset and a simple convolution network can also understand the same. So I would like to know ...
- 111
0
votes
2
answers
456
views
Help with deep Q learning for 2048 game getting stuck
I am having trouble making a reinforcement algorithm than can win the 2048 game.
I have tried with deep Q (which I think is the simplest algorithm that should be able to learn a winning strategy).
...
- 207
3
votes
1
answer
2k
views
Why do we need convolutional neural networks instead of feed-forward neural networks?
Why do we need convolutional neural networks instead of feed-forward neural networks?
What is the significance of a CNN? Even a feed-forward neural network will able to solve the image classification ...
- 1,613
2
votes
1
answer
208
views
What are the most common feedforward neural networks?
What are the most common feedforward neural networks? What kind of inputs do they receive? For example, do they receive binary numbers, real numbers, vectors, or matrics? Is there such a taxonomy?
- 29
3
votes
1
answer
100
views
Accuracy dropped when I ran the program the second time
I was following a tutorial about Feed-Forward Networks and wrote this code for a simple FFN :
...
- 31