Questions tagged [feedforward-neural-networks]

For questions related to feedforward neural networks (FFNNs), which are also sometimes called multilayer perceptrons, but these two expressions may not always be interchangeable.

Filter by
Sorted by
Tagged with
1 vote
1 answer
44 views

Find maxima and minima of feed forward neural network given interval inputs [closed]

I have a feed forward neural network defined in pytorch as follows: ...
  • 13
0 votes
2 answers
70 views

Are neural networks a strict special case of a transformer?

Since transformers contain a neural network, are they a strict generalisation of standard feedforward neural networks? In what ways can transformers be interpreted as a generalisation and abstraction ...
0 votes
0 answers
35 views

Combining loss function while training with another objective function

I would like to train a ReLU neural network minimizing an objective function that looks like this: $$L(W) + \eta + 1_{S}(\eta,W)$$ where $W$ is the set of weight matrices, $L(W)$ is a custom loss ...
  • 111
0 votes
0 answers
37 views

How can I do a time and space complexity comparison of a graph convolutional network vs. a feed forward network / MLP

How can I make a time and space complexity comparison of a graph convolutional network vs a feed-forward network / MLP? Does anybody have an idea how I can compare them?
1 vote
1 answer
93 views

What type of neural network has an unorganized structure?

I am looking for a network that has an unorganized structure like this, is feed-forward, does not have back-propagation functionality, and is trained with a genetic algorithm. What would I be looking ...
  • 17
0 votes
0 answers
22 views

Why would the training loss consistently increase over many epochs?

I am getting a very strange learning curve when I try training a neural network which I am not able to explain. I have never seen a learning curve that looks like this when it's a very simple and ...
  • 101
1 vote
2 answers
46 views

What are all the possible usages of 'multilayer perceptron'?

The term 'multilayer perceptron' has been used in literature in various ways in the literature. I am presenting some of them below As a feed-forward neural network [1]. As a fully connected feed-...
  • 3,391
0 votes
3 answers
113 views

How to deal with an unbalanced dataset?

I'm constructing a feed forward neural network that predicts whether a patient will get a stroke or not. However, my dataset is very unbalanced. Out of 5111 rows, 250 contain patients that have had a ...
0 votes
0 answers
57 views

How to interpret Transformer output

In this (https://towardsdatascience.com/a-detailed-guide-to-pytorchs-nn-transformer-module-c80afbc9ffb1) article the author says, that the output of the ...
1 vote
1 answer
272 views

Is "node embedding" in GNN analogous to "hidden layer" of FFN?

So in Graph Neural Network (GNN) we have node embeddings which is a feature vector that describes the node, is it analogous to hidden layer of Artificial neural network such as feed-forward neural ...
  • 135
2 votes
2 answers
420 views

How can the input order of pairs into a neural network not matter (i.e. symmetry)?

Let me explain, suppose we are building a neural network that predicts if two items are similar or not. This is a classification task with hard labels (0, 1) of examples of similar and dissimilar ...
  • 59
0 votes
1 answer
80 views

How many layers and neurons in a FFNN do I need to make it equivalent to a CNN?

I started to learn machine learning early, and I studied the convolutional neural network and its ability to understand images and how it helps to reduce the number of parameters that need to be tuned....
1 vote
0 answers
44 views

Can I help my neural network if I know the sign of the relationships between inputs and outputs

I am attempting to train a neural network where I can say the following: For most inputs, I know the sign of the relationship between that input and several specific outputs. I.e. whatever set of ...
  • 111
0 votes
1 answer
119 views

How to uniquely associate a directed graph with a feedforward neural network?

I want to write an algorithm that returns a unique directed graph (an adjacency matrix) that represents the structure of a given feedforward neural network (FNN). My idea is to deconstruct the FNN ...
1 vote
0 answers
98 views

Does there exist functions for which the necessary number of nodes in a shallow neural network tends to infinity as approximation error tends to 0?

The Universal Approximation Theorem states (roughly) that any continuous function can be approximated to within an arbitrary precision $\varepsilon>0$ by a feedforward neural network with one ...
2 votes
1 answer
125 views

Closed networks vs Networks with a removed delay to predict new data

I've come across two types of neural networks to predict, both from Matlab, the closed structure and the net that removes one delay to find new data. From Matlab's app generated scripts we see: % ...
1 vote
0 answers
43 views

ReLU function converging to local optimum in one case and diverging in the other one

I implemented a simple neural network with 1 hidden layer. I used ReLU as activation function for the hidden layer and the output layer just uses the linear function. To check my implementation I ...
0 votes
1 answer
77 views

About the choice of the activation functions in the Multilayer Perceptron, and on what does this depends?

I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are ...
2 votes
1 answer
302 views

Why is the backpropagation algorithm used to train the multilayer perceptron?

I've read in the book Neural Network Design, by Martin Hagan et al. (chapter 11), that, to train the feed-forward neural network (aka multilayer perceptron), one uses the backpropagation algorithm. ...
1 vote
1 answer
61 views

Issue with graphical interpretation of the universal approximation theorem

This article attempts to provide a graphical justification of the universal approximation theorem. It succeeds in showing that a linear combination of two sigmoids can produce essentially a bounded ...
1 vote
1 answer
68 views

Can RNNs get inputs and produce outputs similar to the inputs and outputs of FFNNs?

RNN and LSTM models have many architectures that can be modified. We can also compose their input and output data. However, in the examples that I found on the web, the inputs and outputs of RNNs/...
  • 45
1 vote
1 answer
216 views

What are examples of good free books that cover the back-propagation algorithm?

What are examples of good free books that cover the back-propagation used to train multilayer perceptrons? I've just started to learn about artificial neural networks, so I'm looking for books that ...
2 votes
0 answers
162 views

Bias gradient of layer before batch normalization always zero

From the original paper and this post we have that batch normalization backpropagation can be formulated as I'm interested in the derivative of the previous layer outputs $x_i=\sigma(w X_i+b)$ with ...
6 votes
1 answer
841 views

Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?

Why does a vanilla feedforward neural network only accept a fixed input size, while RNNs are capable of taking a series of inputs with no predetermined limit on the size? Can anyone elaborate on this ...
  • 63
5 votes
2 answers
234 views

Is it really possible to create the "Perfect Cylinder" used in Universal Approximation Theorem for 1-hidden layer Neural Network?

There are proofs for the universal approximation theorem with just 1 hidden layer. The proof goes like this: Create a "bump" function using 2 neurons. Create (infinitely) many of these ...
  • 133
6 votes
1 answer
621 views

How to express a fully connected neural network succintly using linear algebra?

I'm currently reading the paper Federated Learning with Matched Averaging (2020), where the authors claim: A basic fully connected (FC) NN can be formulated as: $\hat{y} = \sigma(xW_1)W_2$ [...] ...
1 vote
1 answer
2k views

What is the space complexity for training a neural network using back-propagation?

Suppose that a simple feedforward neural network (FFNN) contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_l$ nodes in each layer. What is the space complexity to train this FFNN ...
1 vote
0 answers
176 views

Why should variance(output) equal variance(input) in Xavier Initialisation?

In a lot of explanations online for Xavier Initialization, I see the following: With each passing layer, we want the variance to remain the same. This helps us keep the signal from exploding to a ...
2 votes
2 answers
266 views

How to tell a neural network that: "your i-th input is special"

Assume that I have a fully connected network that takes in a vector containing 1025 elements. First 1024 elements are related to ...
  • 420
1 vote
2 answers
671 views

How is the error calculated with multiple output neurons in the neural network?

Machine Learning books generally explains that the error calculated for a given sample $i$ is: $e_i = y_i - \hat{y_i}$ Where $\hat{y}$ is the target output and $y$ is the actual output given by the ...
  • 111
2 votes
1 answer
437 views

Can the hidden layer prior to the ouput layer have less hidden units than the output layer?

I attended an introductory class about neural network and I had a question regarding how to choose the number of hidden units per hidden layer. I remember that the Professor saying that there is no ...
1 vote
0 answers
405 views

How to use residual learning applied to fully connected networks?

Is there any reason why skip connections would not provide the same benefits to fully connected layers as it does for convolutional? I've read the ResNet paper and it says that the applications should ...
1 vote
0 answers
30 views

What's the difference between RNNs and Feed Forward Neural Networks if a fixed size vector can preserve sequential information?

I was watching a Youtube video in which the problem of trying to predict the last word in a sentence was posed. The sentence was "I took my cat for a" and the last word was "walk"....
1 vote
1 answer
66 views

What's a good neural network for this problem?

I am very new to the field of AI so please bear with me. Say there is a dice with three sides, -1,0 and 1, and I want to predict which side it lands on (so only one output is needed I guess). The ...
2 votes
1 answer
105 views

What is the Preferred Mathematical Representation for a Forward Pass in a Neural Network?

I know this may be a question of semantics but I always see different articles explain forward pass slightly different. e.g. Sometimes they represent a forward pass to a hidden layer in a standard ...
0 votes
2 answers
416 views

Why isn't the loss of my neural network reduced after 2500 iterations?

I have developed a basic feedforward neural network from scratch to classify whether image is of cat or not cat. It works fine, but after 2500 iterations, my cost function is not reducing properly. ...
1 vote
2 answers
76 views

Can we achieve what a CNN can do with just a normal neural network?

When I was learning about neural networks, I saw that a complex neural network can understand the MNIST dataset and a simple convolution network can also understand the same. So I would like to know ...
-1 votes
2 answers
327 views

Help with deep Q learning for 2048 game getting stuck

I am having trouble making a reinforcement algorithm than can win the 2048 game. I have tried with deep Q (which I think is the simplest algorithm that should be able to learn a winning strategy). ...
3 votes
1 answer
878 views

Why do we need convolutional neural networks instead of feed-forward neural networks?

Why do we need convolutional neural networks instead of feed-forward neural networks? What is the significance of a CNN? Even a feed-forward neural network will able to solve the image classification ...
  • 1,353
2 votes
1 answer
137 views

What are the most common feedforward neural networks?

What are the most common feedforward neural networks? What kind of inputs do they receive? For example, do they receive binary numbers, real numbers, vectors, or matrics? Is there such a taxonomy?
3 votes
1 answer
93 views

Accuracy dropped when I ran the program the second time

I was following a tutorial about Feed-Forward Networks and wrote this code for a simple FFN : ...
8 votes
2 answers
2k views

What exactly is averaged when doing batch gradient descent?

I have a question about how the averaging works when doing mini-batch gradient descent. I think I now understood the general gradient descent algorithm, but only for online learning. When doing mini-...
  • 405
2 votes
2 answers
143 views

What are standard datasets for fully connected neural networks?

I am looking for datasets that are used as a testing standard in the fully connected neural networks (FCNN). For example, in the image recognition and CNN, CIFAR datasets are used in most of the ...
1 vote
0 answers
42 views

How do we minimize loss for a single neuron with a feedback?

Suppose we had a series of single-dimensional data points $X = \{x_1, x_2, \dots, x_n \}$, where $n$ is the number of data points and there corresponding output values $T = \{t_1, t_2, \dots, t_n \}$. ...
1 vote
0 answers
27 views

Can denoising auto-encoders be convolutional and fully connected?

I have been reading lately on autoencoders a lot. I just wanted to summarize my understanding of denoising autoencoders. As far as I understand they can be Fully connected (in which case, they will ...
  • 111
2 votes
2 answers
210 views

What happens if I train a network for more epochs, without using early stopping?

I have a question about training a neural network for more epochs even after the network has converged without using early stopping criterion. Consider the MNIST dataset and a LeNet 300-100-10 dense ...
  • 205
1 vote
1 answer
80 views

Are there names for neural networks with a well-defined layer or neuron characteristics?

Are there names for neural networks with a well-defined layer or neuron characteristics? For example, a matrix that has the same number of rows and columns is called a square matrix. Is there an ...
  • 210
1 vote
0 answers
29 views

Neural network seems to just figure out the probability of a specific result

I am really new to neural networks, so i was following along with a video series, created by '3blue1brown' on youtube. I created an implementation of the network he explained in c++. I am attempting ...
2 votes
2 answers
128 views

What is the name of this neural network architecture with layers that are also connected to non-neighbouring layers?

Consider a feedforward neural network. Suppose you have a layer of inputs, which is feedforward to a hidden layer, and feedforward both the input and hidden layers to an output layer. Is there a name ...
2 votes
0 answers
39 views

How do I determine the best neural network architecture for a problem with 3 inputs and 12 outputs?

This post continues the topic in the following post: Is it possible to train a neural network with 3 inputs and 12 outputs?. I conducted several experiments in MATLAB and selected those neural ...
  • 227