Questions tagged [feedforward-neural-networks]

For questions related to feedforward neural networks (FFNNs), which are also sometimes called multilayer perceptrons, but these two expressions may not always be interchangeable.

Filter by
Sorted by
Tagged with
0
votes
1answer
48 views

How many layers and neurons in a FFNN do I need to make it equivalent to a CNN?

I started to learn machine learning early, and I studied the convolutional neural network and its ability to understand images and how it helps to reduce the number of parameters that need to be tuned....
1
vote
0answers
41 views

Can I help my neural network if I know the sign of the relationships between inputs and outputs

I am attempting to train a neural network where I can say the following: For most inputs, I know the sign of the relationship between that input and several specific outputs. I.e. whatever set of ...
0
votes
1answer
46 views

How to uniquely associate a directed graph with a feedforward neural network?

I want to write an algorithm that returns a unique directed graph (an adjacency matrix) that represents the structure of a given feedforward neural network (FNN). My idea is to deconstruct the FNN ...
1
vote
0answers
56 views

Does there exist functions for which the necessary number of nodes in a shallow neural network tends to infinity as approximation error tends to 0?

The Universal Approximation Theorem states (roughly) that any continuous function can be approximated to within an arbitrary precision $\varepsilon>0$ by a feedforward neural network with one ...
2
votes
1answer
95 views

Closed networks vs Networks with a removed delay to predict new data

I've come across two types of neural networks to predict, both from Matlab, the closed structure and the net that removes one delay to find new data. From Matlab's app generated scripts we see: % ...
1
vote
0answers
24 views

ReLU function converging to local optimum in one case and diverging in the other one

I implemented a simple neural network with 1 hidden layer. I used ReLU as activation function for the hidden layer and the output layer just uses the linear function. To check my implementation I ...
0
votes
0answers
18 views

when updating the bias matrix, do we get the total sum of dZ or the sum of the axis of dZ?

I'm currently studying how to implement a neural network from scratch to know how it works, I came across this article: https://www.samsonzhang.com/2020/11/24/understanding-the-math-behind-neural-...
0
votes
1answer
60 views

About the choice of the activation functions in the Multilayer Perceptron, and on what does this depends?

I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are ...
2
votes
1answer
146 views

Why is the backpropagation algorithm used to train the multilayer perceptron?

I've read in the book Neural Network Design, by Martin Hagan et al. (chapter 11), that, to train the feed-forward neural network (aka multilayer perceptron), one uses the backpropagation algorithm. ...
1
vote
1answer
36 views

Issue with graphical interpretation of the universal approximation theorem

This article attempts to provide a graphical justification of the universal approximation theorem. It succeeds in showing that a linear combination of two sigmoids can produce essentially a bounded ...
1
vote
1answer
59 views

Can RNNs get inputs and produce outputs similar to the inputs and outputs of FFNNs?

RNN and LSTM models have many architectures that can be modified. We can also compose their input and output data. However, in the examples that I found on the web, the inputs and outputs of RNNs/...
0
votes
0answers
49 views

Why is the error curve of a neural network trained with MSE to output $\frac{3 I_1 + 5 I_2}{2}$ given inputs $I_1$ and $I_2$ oscillating weirdly?

I just "finished" my first AI program. I programmed in Excel VBA, and I think it works well. I was checking every formula and the whole algorithm several times to make sure every formula is ...
1
vote
1answer
95 views

What are examples of good free books that cover the back-propagation algorithm?

What are examples of good free books that cover the back-propagation used to train multilayer perceptrons? I've just started to learn about artificial neural networks, so I'm looking for books that ...
2
votes
0answers
98 views

Bias gradient of layer before batch normalization always zero

From the original paper and this post we have that batch normalization backpropagation can be formulated as I'm interested in the derivative of the previous layer outputs $x_i=\sigma(w X_i+b)$ with ...
4
votes
1answer
404 views

Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?

Why does a vanilla feedforward neural network only accept a fixed input size, while RNNs are capable of taking a series of inputs with no predetermined limit on the size? Can anyone elaborate on this ...
5
votes
1answer
189 views

Is it really possible to create the "Perfect Cylinder" used in Universal Approximation Theorem for 1-hidden layer Neural Network?

There are proofs for the universal approximation theorem with just 1 hidden layer. The proof goes like this: Create a "bump" function using 2 neurons. Create (infinitely) many of these ...
6
votes
1answer
385 views

How to express a fully connected neural network succintly using linear algebra?

I'm currently reading the paper Federated Learning with Matched Averaging (2020), where the authors claim: A basic fully connected (FC) NN can be formulated as: $\hat{y} = \sigma(xW_1)W_2$ [...] ...
1
vote
1answer
745 views

What is the space complexity for training a neural network using back-propagation?

Suppose that a simple feedforward neural network (FFNN) contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_l$ nodes in each layer. What is the space complexity to train this FFNN ...
1
vote
0answers
74 views

Why should variance(output) equal variance(input) in Xavier Initialisation?

In a lot of explanations online for Xavier Initialization, I see the following: With each passing layer, we want the variance to remain the same. This helps us keep the signal from exploding to a ...
2
votes
2answers
257 views

How to tell a neural network that: "your i-th input is special"

Assume that I have a fully connected network that takes in a vector containing 1025 elements. First 1024 elements are related to ...
1
vote
2answers
204 views

How is the error calculated with multiple output neurons in the neural network?

Machine Learning books generally explains that the error calculated for a given sample $i$ is: $e_i = y_i - \hat{y_i}$ Where $\hat{y}$ is the target output and $y$ is the actual output given by the ...
2
votes
1answer
274 views

Can the hidden layer prior to the ouput layer have less hidden units than the output layer?

I attended an introductory class about neural network and I had a question regarding how to choose the number of hidden units per hidden layer. I remember that the Professor saying that there is no ...
1
vote
0answers
198 views

How to use residual learning applied to fully connected networks?

Is there any reason why skip connections would not provide the same benefits to fully connected layers as it does for convolutional? I've read the ResNet paper and it says that the applications should ...
1
vote
0answers
24 views

What's the difference between RNNs and Feed Forward Neural Networks if a fixed size vector can preserve sequential information?

I was watching a Youtube video in which the problem of trying to predict the last word in a sentence was posed. The sentence was "I took my cat for a" and the last word was "walk"....
1
vote
1answer
62 views

What's a good neural network for this problem?

I am very new to the field of AI so please bear with me. Say there is a dice with three sides, -1,0 and 1, and I want to predict which side it lands on (so only one output is needed I guess). The ...
2
votes
1answer
69 views

What is the Preferred Mathematical Representation for a Forward Pass in a Neural Network?

I know this may be a question of semantics but I always see different articles explain forward pass slightly different. e.g. Sometimes they represent a forward pass to a hidden layer in a standard ...
0
votes
2answers
191 views

Why isn't the loss of my neural network reduced after 2500 iterations?

I have developed a basic feedforward neural network from scratch to classify whether image is of cat or not cat. It works fine, but after 2500 iterations, my cost function is not reducing properly. ...
1
vote
2answers
72 views

Can we achieve what a CNN can do with just a normal neural network?

When I was learning about neural networks, I saw that a complex neural network can understand the MNIST dataset and a simple convolution network can also understand the same. So I would like to know ...
0
votes
2answers
272 views

Help with deep Q learning for 2048 game getting stuck

I am having trouble making a reinforcement algorithm than can win the 2048 game. I have tried with deep Q (which I think is the simplest algorithm that should be able to learn a winning strategy). ...
3
votes
1answer
337 views

Why do we need convolutional neural networks instead of feed-forward neural networks?

Why do we need convolutional neural networks instead of feed-forward neural networks? What is the significance of a CNN? Even a feed-forward neural network will able to solve the image classification ...
3
votes
1answer
93 views

What are the most common feedforward neural networks?

What are the most common feedforward neural networks? What kind of inputs do they receive? For example, do they receive binary numbers, real numbers, vectors, or matrics? Is there such a taxonomy?
3
votes
1answer
81 views

Accuracy dropped when I ran the program the second time

I was following a tutorial about Feed-Forward Networks and wrote this code for a simple FFN : ...
6
votes
2answers
971 views

What exactly is averaged when doing batch gradient descent?

I have a question about how the averaging works when doing mini-batch gradient descent. I think I now understood the general gradient descent algorithm, but only for online learning. When doing mini-...
2
votes
2answers
97 views

What are standard datasets for fully connected neural networks?

I am looking for datasets that are used as a testing standard in the fully connected neural networks (FCNN). For example, in the image recognition and CNN, CIFAR datasets are used in most of the ...
1
vote
0answers
34 views

How do we minimize loss for a single neuron with a feedback?

Suppose we had a series of single-dimensional data points $X = \{x_1, x_2, \dots, x_n \}$, where $n$ is the number of data points and there corresponding output values $T = \{t_1, t_2, \dots, t_n \}$. ...
1
vote
0answers
26 views

Can denoising auto-encoders be convolutional and fully connected?

I have been reading lately on autoencoders a lot. I just wanted to summarize my understanding of denoising autoencoders. As far as I understand they can be Fully connected (in which case, they will ...
1
vote
2answers
77 views

What happens if I train a network for more epochs, without using early stopping?

I have a question about training a neural network for more epochs even after the network has converged without using early stopping criterion. Consider the MNIST dataset and a LeNet 300-100-10 dense ...
1
vote
1answer
72 views

Are there names for neural networks with a well-defined layer or neuron characteristics?

Are there names for neural networks with a well-defined layer or neuron characteristics? For example, a matrix that has the same number of rows and columns is called a square matrix. Is there an ...
1
vote
0answers
26 views

Neural network seems to just figure out the probability of a specific result

I am really new to neural networks, so i was following along with a video series, created by '3blue1brown' on youtube. I created an implementation of the network he explained in c++. I am attempting ...
2
votes
2answers
86 views

What is the name of this neural network architecture with layers that are also connected to non-neighbouring layers?

Consider a feedforward neural network. Suppose you have a layer of inputs, which is feedforward to a hidden layer, and feedforward both the input and hidden layers to an output layer. Is there a name ...
2
votes
0answers
34 views

How do I determine the best neural network architecture for a problem with 3 inputs and 12 outputs?

This post continues the topic in the following post: Is it possible to train a neural network with 3 inputs and 12 outputs?. I conducted several experiments in MATLAB and selected those neural ...
2
votes
1answer
181 views

How to make DNN learn multiplication/division?

A single neuron with 2 weights and identity activation can learn addition/subtraction as the 2 weights will converge to 1 and 1 (addition), or 1 and -1 (subtraction). However, for multiplication and ...
4
votes
0answers
54 views

Given an input $x \in R^{1\times d}$ and a network with $s$ hidden layers, is the time complexity of the forward pass $O(d^{2}s)$? [duplicate]

I have a neural network that takes as an input a vector of $x \in R^{1\times d}$ with $s$ hidden layers and each layer has $d$ neurons (including the output layer). If I understand correctly the ...
1
vote
0answers
141 views

Is this TensorFlow implementation of partial derivative of the cost with respect to the bias correct?

I have a neural network for MNIST classification which I am hard coding using TensorFlow 2.0. The neural network has an input layer consisting of 784 neurons (28 * 28), one hidden layer having "...
3
votes
1answer
198 views

What kind of data structures are needed to efficiently do back-propagation in a feedforward neural network?

In a feed-forward neural network, in order to efficiently do backpropagation, what kind of data structure is needed? I know the weights can just be stored in an array, and you need pointers of some ...
1
vote
0answers
20 views

Self-organizing map using weighted non-euclidean distance to minimize variance of predictions

Let's say I have a dataset, each item/row of which has $\mathit{X + 1}$ characteristics where the last characteristic (i.e., the $\mathit{1}$) represents the some value I want to predict, $\mathit{Y}$,...
8
votes
2answers
1k views

Why use a recurrent neural network over a feedforward neural network for sequence prediction?

If recurrent neural networks (RNNs) are used to capture prior information, couldn't the same thing be achieved by a feedforward neural network (FFNN) or multi-layer perceptron (MLP) where the inputs ...
0
votes
1answer
88 views

How can we print weights per iteration in a simple feed forward MLP for an specific class?

im working on a project in which I have to make a multi-layer perceptron with two hidden layers with 3 nodes in each. The target value in my data contains 8 unique values/classes. One of the tasks ...
1
vote
0answers
21 views

Matrix-output for FFNN?

Turns out that it looks like I will be approximating a 100x10 matrix in my project thesis. II have the following equation $y = Dx$, where $y$ is $(100 \times 1)$, $D$ is $100 \times 10$ and $x$ is $...
15
votes
2answers
5k views

Why would you implement the position-wise feed-forward network of the transformer with convolution layers?

The Transformer model introduced in "Attention is all you need" by Vaswani et al. incorporates a so-called position-wise feed-forward network (FFN): In addition to attention sub-layers, each of the ...