Questions tagged [feedforward-neural-network]

For questions related to feedforward neural networks (FFNNs), which are also sometimes called multilayer perceptrons, but these two expressions may not always be interchangeable.

Filter by
Sorted by
Tagged with
1
vote
1answer
54 views
+100

Closed networks vs Networks with a removed delay to predict new data

I've come across two types of neural networks to predict, both from Matlab, the closed structure and the net that removes one delay to find new data. From Matlab's app generated scripts we see: % ...
3
votes
1answer
80 views

Accuracy dropped when I ran the program the second time

I was following a tutorial about Feed-Forward Networks and wrote this code for a simple FFN : ...
2
votes
2answers
79 views

Attempting to solve a optical character recognition task using a feed-forward network

I am doing some experimentation on neural networks, and for that I am trying to program a plain OCR task. I have learned CNNs are the best choice ,but for the time being and due to my inexperience, I ...
1
vote
1answer
67 views

How does a single neuron in hidden layer affect training accuracy [duplicate]

I'm currently a student learning about AI Networks. I've came across a statement in one of my Professor's books that a FFBP (Feed-Forward Back-Propagation) Neural Network with a single hidden layer ...
0
votes
2answers
177 views

Why isn't the loss of my neural network reduced after 2500 iterations?

I have developed a basic feedforward neural network from scratch to classify whether image is of cat or not cat. It works fine, but after 2500 iterations, my cost function is not reducing properly. ...
2
votes
0answers
20 views

ReLU function converging to local optimum in one case and diverging in the other one

I implemented a simple neural network with 1 hidden layer. I used ReLU as activation function for the hidden layer and the output layer just uses the linear function. To check my implementation I ...
1
vote
3answers
1k views

How to create Partially Connected NNs with prespecified connections using Tensorflow?

I'd like to implement a partially connected neural network with ~3 to 4 hidden layers (a sparse deep neural network?) where I can specify which node connects to which node from the previous/next layer....
0
votes
0answers
11 views

when updating the bias matrix, do we get the total sum of dZ or the sum of the axis of dZ?

I'm currently studying how to implement a neural network from scratch to know how it works, I came across this article: https://www.samsonzhang.com/2020/11/24/understanding-the-math-behind-neural-...
1
vote
1answer
460 views

What is the space complexity for training a neural network using back-propagation?

Suppose that a simple feedforward neural network (FFNN) contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_l$ nodes in each layer. What is the space complexity to train this FFNN ...
2
votes
2answers
246 views

How to tell a neural network that: "your i-th input is special"

Assume that I have a fully connected network that takes in a vector containing 1025 elements. First 1024 elements are related to ...
2
votes
1answer
122 views

Why is the Backpropagation algorithm used to train the Multilayer Perceptron?

I've read on the book NND by Martin Hagan et al (chapter 11), that to train the feed-forward neural network: Multilayer Perceptron one uses the Backpropagation algorithm. Why this algorithm? Could ...
2
votes
1answer
346 views

Maximum number of neurons in a layer given number of neurons in previous layer

Consider an extremely complicated feed-forward neural network training example but with no need of computational efficiency or limiting of processing time. What is the maximum number of hidden ...
0
votes
1answer
56 views

About the choice of the activation functions in the Multilayer Perceptron, and on what does this depends?

I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are ...
4
votes
1answer
319 views

Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?

Why does a vanilla feedforward neural network only accept a fixed input size, while RNNs are capable of taking a series of inputs with no predetermined limit on the size? Can anyone elaborate on this ...
1
vote
1answer
47 views

Can RNNs get inputs and produce outputs similar to the inputs and outputs of FFNNs?

RNN and LSTM models have many interesting architectures that can be modified in various ways. We can also compose their input and output data in quite interesting ways. However, in the examples that I ...
1
vote
1answer
35 views

Issue with graphical interpretation of the universal approximation theorem

This article attempts to provide a graphical justification of the universal approximation theorem. It succeeds in showing that a linear combination of two sigmoids can produce essentially a bounded ...
1
vote
1answer
78 views

What are examples of good free books that cover the back-propagation algorithm?

What are examples of good free books that cover the back-propagation used to train multilayer perceptrons? I've just started to learn about artificial neural networks, so I'm looking for books that ...
0
votes
0answers
49 views

Why is the error curve of a neural network trained with MSE to output $\frac{3 I_1 + 5 I_2}{2}$ given inputs $I_1$ and $I_2$ oscillating weirdly?

I just "finished" my first AI program. I programmed in Excel VBA, and I think it works well. I was checking every formula and the whole algorithm several times to make sure every formula is ...
2
votes
0answers
85 views

Bias gradient of layer before batch normalization always zero

From the original paper and this post we have that batch normalization backpropagation can be formulated as I'm interested in the derivative of the previous layer outputs $x_i=\sigma(w X_i+b)$ with ...
4
votes
1answer
162 views

Is it really possible to create the "Perfect Cylinder" used in Universal Approximation Theorem for 1-hidden layer Neural Network?

There are proofs for the universal approximation theorem with just 1 hidden layer. The proof goes like this: Create a "bump" function using 2 neurons. Create (infinitely) many of these ...
0
votes
0answers
18 views

Neural Network trains towards 1 despite target

So I'm trying to make my first neural network and have just finished my back propagation functions. I got the algebra from brilliant and thought I'd understood it, but my bug proves otherwise. The bug ...
6
votes
1answer
356 views

How to express a fully connected neural network succintly using linear algebra?

I'm currently reading the paper Federated Learning with Matched Averaging (2020), where the authors claim: A basic fully connected (FC) NN can be formulated as: $\hat{y} = \sigma(xW_1)W_2$ [...] ...
3
votes
1answer
113 views

Could new training pictures destroy the trained weights of the neural network?

Let's say an image has 28*28 pixels, which leads to 784 input nodes in a feed-forward neural network. If an image can be classified into 1 of 10 numbers (e.g. MNIST), there are 10 output nodes. We ...
2
votes
1answer
274 views

Why do we need convolutional neural networks instead of feed-forward neural networks?

Why do we need convolutional neural networks instead of feed-forward neural networks? What is the significance of a CNN? Even a feed-forward neural network will able to solve the image classification ...
1
vote
0answers
48 views

Why should variance(output) equal variance(input) in Xavier Initialisation?

In a lot of explanations online for Xavier Initialization, I see the following: With each passing layer, we want the variance to remain the same. This helps us keep the signal from exploding to a ...
5
votes
2answers
753 views

What exactly is averaged when doing batch gradient descent?

I have a question about how the averaging works when doing mini-batch gradient descent. I think I now understood the general gradient descent algorithm, but only for online learning. When doing mini-...
0
votes
0answers
10 views

Is there a framework or method that would help visualise inner workings of a feedforward neural network?

I wonder if there is some framework or method to help visualising inner workings of a feedforward deep neural network? What I mean by this is something similar to what is being done with CNNs where we ...
1
vote
2answers
138 views

How is the error calculated with multiple output neurons in the neural network?

Machine Learning books generally explains that the error calculated for a given sample $i$ is: $e_i = y_i - \hat{y_i}$ Where $\hat{y}$ is the target output and $y$ is the actual output given by the ...
2
votes
1answer
255 views

Can the hidden layer prior to the ouput layer have less hidden units than the output layer?

I attended an introductory class about neural network and I had a question regarding how to choose the number of hidden units per hidden layer. I remember that the Professor saying that there is no ...
2
votes
1answer
60 views

What is the Preferred Mathematical Representation for a Forward Pass in a Neural Network?

I know this may be a question of semantics but I always see different articles explain forward pass slightly different. e.g. Sometimes they represent a forward pass to a hidden layer in a standard ...
9
votes
1answer
3k views

What is the time complexity of the forward pass algorithm of a feedforward neural network?

How do I determine the time complexity of the forward pass algorithm of a feedforward neural network? How many multiplications are done to generate the output?
1
vote
0answers
153 views

How to use residual learning applied to fully connected networks?

Is there any reason why skip connections would not provide the same benefits to fully connected layers as it does for convolutional? I've read the ResNet paper and it says that the applications should ...
1
vote
2answers
469 views

Why are RNNs better than MLPs at predicting time series data?

Understandably RNNs are very good at solving problems involving audio, video and text processing due to the arbitrary input's length of this sort of data. What I don't understand is why RNNs are also ...
1
vote
0answers
24 views

What's the difference between RNNs and Feed Forward Neural Networks if a fixed size vector can preserve sequential information?

I was watching a Youtube video in which the problem of trying to predict the last word in a sentence was posed. The sentence was "I took my cat for a" and the last word was "walk"....
1
vote
1answer
59 views

What's a good neural network for this problem?

I am very new to the field of AI so please bear with me. Say there is a dice with three sides, -1,0 and 1, and I want to predict which side it lands on (so only one output is needed I guess). The ...
0
votes
0answers
35 views

What kind of ANN should I use for this problem?

I want to develop a neural network for my data, but I have no idea if my problem should be solved with curve fitting, classification or maybe something else. Here is my problem. My input data are some ...
2
votes
1answer
82 views

What are the most common feedforward neural networks?

What are the most common feedforward neural networks? What kind of inputs do they receive? For example, do they receive binary numbers, real numbers, vectors, or matrics? Is there such a taxonomy?
15
votes
2answers
4k views

Why would you implement the position-wise feed-forward network of the transformer with convolution layers?

The Transformer model introduced in "Attention is all you need" by Vaswani et al. incorporates a so-called position-wise feed-forward network (FFN): In addition to attention sub-layers, each of the ...
0
votes
2answers
241 views

Help with deep Q learning for 2048 game getting stuck

I am having trouble making a reinforcement algorithm than can win the 2048 game. I have tried with deep Q (which I think is the simplest algorithm that should be able to learn a winning strategy). ...
1
vote
2answers
67 views

Can we achieve what a CNN can do with just a normal neural network?

When I was learning about neural networks, I saw that a complex neural network can understand the MNIST dataset and a simple convolution network can also understand the same. So I would like to know ...
12
votes
2answers
2k views

What is a recurrent neural network?

Surprisingly, this wasn't asked before - at least I didn't find anything besides some vaguely related questions. So, what is a recurrent neural network, and what are their advantages over regular (or ...
2
votes
0answers
33 views

Why do we need recurrent neural networks instead of feed-forward neural networks? [duplicate]

Why do we need recurrent neural networks instead of feed-forward neural networks? What are the advantages of RNNs compared with FFNNs?
0
votes
0answers
116 views

How can feedforward neural networks act as contraction maps?

In graph neural networks, the Banach fixed-point theorem and Jacobi method it is described that the transition from one state to another be defined by a contraction map with a fixed-point. The autor ...
2
votes
2answers
86 views

What are standard datasets for fully connected neural networks?

I am looking for datasets that are used as a testing standard in the fully connected neural networks (FCNN). For example, in the image recognition and CNN, CIFAR datasets are used in most of the ...
17
votes
2answers
297 views

How do I decide the optimal number of layers for a neural network?

How do I decide the optimal number of layers for a neural network (feedforward or recurrent)?
1
vote
0answers
34 views

How do we minimize loss for a single neuron with a feedback?

Suppose we had a series of single-dimensional data points $X = \{x_1, x_2, \dots, x_n \}$, where $n$ is the number of data points and there corresponding output values $T = \{t_1, t_2, \dots, t_n \}$. ...
1
vote
0answers
26 views

Can denoising auto-encoders be convolutional and fully connected?

I have been reading lately on autoencoders a lot. I just wanted to summarize my understanding of denoising autoencoders. As far as I understand they can be Fully connected (in which case, they will ...
7
votes
1answer
643 views

Why aren't there neural networks that connect the output of each layer to all next layers?

Why aren't there neural networks that connect the output of each layer to all next layers? For example, the output of layer 1 would be fed to the input of layers 2, 3, 4, etc. Beyond computational ...
1
vote
2answers
74 views

What happens if I train a network for more epochs, without using early stopping?

I have a question about training a neural network for more epochs even after the network has converged without using early stopping criterion. Consider the MNIST dataset and a LeNet 300-100-10 dense ...
1
vote
0answers
76 views

Why would my neural network run faster on my laptop than on my university's supercomputer?

I am trying to get my neural network running on my university's supercomputer in order to decrease its runtime (not for training, for testing - feedforward runs only). However, the Matlab function I ...