# Questions tagged [feedforward-neural-network]

For questions related to feedforward neural networks (FFNNs), which are also sometimes called multilayer perceptrons, but these two expressions may not always be interchangeable.

80 questions
Filter by
Sorted by
Tagged with
40 views

### Why is the Backpropagation algorithm used to train the Multilayer Perceptron?

I've read on the book NND by Martin Hagan et al (chapter 11 IIRC), that to train the feed-forward neural network: Multilayer Perceptron one uses the Backpropagation algorithm. Why? Why not another?
33 views

### Issue with graphical interpretation of the universal approximation theorem

This article attempts to provide a graphical justification of the universal approximation theorem. It succeeds in showing that a linear combination of two sigmoids can produce essentially a bounded ...
40 views

### Can RNNs get inputs and produce outputs similar to the inputs and outputs of FFNNs?

RNN and LSTM models have many interesting architectures that can be modified in various ways. We can also compose their input and output data in quite interesting ways. However, in the examples that I ...
49 views

### Why is the error curve of a neural network trained with MSE to output $\frac{3 I_1 + 5 I_2}{2}$ given inputs $I_1$ and $I_2$ oscillating weirdly?

I just "finished" my first AI program. I programmed in Excel VBA, and I think it works well. I was checking every formula and the whole algorithm several times to make sure every formula is ...
55 views

### What are examples of good free books that cover the back-propagation algorithm?

What are examples of good free books that cover the back-propagation used to train multilayer perceptrons? I've just started to learn about artificial neural networks, so I'm looking for books that ...
69 views

### Bias gradient of layer before batch normalization always zero

From the original paper and this post we have that batch normalization backpropagation can be formulated as I'm interested in the derivative of the previous layer outputs $x_i=\sigma(w X_i+b)$ with ...
201 views

### Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?

Why does a vanilla feedforward neural network only accept a fixed input size, while RNNs are capable of taking a series of inputs with no predetermined limit on the size? Can anyone elaborate on this ...
18 views

### Neural Network trains towards 1 despite target

So I'm trying to make my first neural network and have just finished my back propagation functions. I got the algebra from brilliant and thought I'd understood it, but my bug proves otherwise. The bug ...
156 views

### Is it really possible to create the “Perfect Cylinder” used in Universal Approximation Theorem for 1-hidden layer Neural Network?

There are proofs for the universal approximation theorem with just 1 hidden layer. The proof goes like this: Create a "bump" function using 2 neurons. Create (infinitely) many of these ...
335 views

### How to express a fully connected neural network succintly using linear algebra?

I'm currently reading the paper Federated Learning with Matched Averaging (2020), where the authors claim: A basic fully connected (FC) NN can be formulated as: $\hat{y} = \sigma(xW_1)W_2$ [...] ...
159 views

### What is the space complexity for training a neural network using back-propagation?

Suppose that a simple feedforward neural network (FFNN) contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_l$ nodes in each layer. What is the space complexity to train this FFNN ...
33 views

### Why should variance(output) equal variance(input) in Xavier Initialisation?

In a lot of explanations online for Xavier Initialization, I see the following: With each passing layer, we want the variance to remain the same. This helps us keep the signal from exploding to a ...
9 views

### Is there a framework or method that would help visualise inner workings of a feedforward neural network?

I wonder if there is some framework or method to help visualising inner workings of a feedforward deep neural network? What I mean by this is something similar to what is being done with CNNs where we ...
234 views

### How to tell a neural network that: “your i-th input is special”

Assume that I have a fully connected network that takes in a vector containing 1025 elements. First 1024 elements are related to ...
75 views

### How is the error calculated with multiple output neurons in the neural network?

Machine Learning books generally explains that the error calculated for a given sample $i$ is: $e_i = y_i - \hat{y_i}$ Where $\hat{y}$ is the target output and $y$ is the actual output given by the ...
235 views

### Can the hidden layer prior to the ouput layer have less hidden units than the output layer?

I attended an introductory class about neural network and I had a question regarding how to choose the number of hidden units per hidden layer. I remember that the Professor saying that there is no ...
81 views

### How to use residual learning applied to fully connected networks?

Is there any reason why skip connections would not provide the same benefits to fully connected layers as it does for convolutional? I've read the ResNet paper and it says that the applications should ...
23 views

### How to enforce non-negative outputs in a feed-forward neural network in tensorflow after appropriate scaling?

I am using tf.keras for a regression problem. I am building a simple feed-forward neural network with one hidden layer and tanh activation. (I started with the neural network toolbox as a part of ...
19 views

### What's the difference between RNNs and Feed Forward Neural Networks if a fixed size vector can preserve sequential information?

I was watching a Youtube video in which the problem of trying to predict the last word in a sentence was posed. The sentence was "I took my cat for a" and the last word was "walk"....
58 views

### What's a good neural network for this problem?

I am very new to the field of AI so please bear with me. Say there is a dice with three sides, -1,0 and 1, and I want to predict which side it lands on (so only one output is needed I guess). The ...
34 views

### What kind of ANN should I use for this problem?

I want to develop a neural network for my data, but I have no idea if my problem should be solved with curve fitting, classification or maybe something else. Here is my problem. My input data are some ...
58 views

### What is the Preferred Mathematical Representation for a Forward Pass in a Neural Network?

I know this may be a question of semantics but I always see different articles explain forward pass slightly different. e.g. Sometimes they represent a forward pass to a hidden layer in a standard ...
35 views

### How to interpret the final neural network generated in NEAT-Python?

I made a simple obstacle collision game in Python and made a neural network which will play the game using the NEAT (NeuroEvolution of Augmenting Topologies) and I got a final network as follows -> ...
49 views

### ALL OUTPUTS for my neural network return SAME as [0.0, 0.0, 0.0] all of time. Trying to make an AI that plays the Snake Game in Python using NEAT

I am trying to make an AI that plays the Snake Game using the NEAT library in Python. I have most things figured out. I have chosen 6 inputs for my neural network -> Is it clear in forward ...
39 views

### Using a neural network in a microcontroller to recognize air-written letters and numbers

Recently I finished a project which combined stm32f030r8 microcontroller and MPU9250 sensor to create a system which would detect orientation on all 3 axis using a combination of accel data, gyro data ...
167 views

### Why isn't the loss of my neural network reduced after 2500 iterations?

I have developed a basic feedforward neural network from scratch to classify whether image is of cat or not cat. It works fine, but after 2500 iterations, my cost function is not reducing properly. ...
31 views

### How can Kneser-Ney Smoothing be integrated into a neural language model?

I found a paper titled Multimodal representation: Kneser-Ney Smoothing/Skip-Gram based neural language model. I am curious about how can Kneser-Ney Smoothing technique be integrated into a feed-...
64 views

### Can we achieve what a CNN can do with just a normal neural network?

When I was learning about neural networks, I saw that a complex neural network can understand the MNIST dataset and a simple convolution network can also understand the same. So I would like to know ...
175 views

### Help with deep Q learning for 2048 game getting stuck

I am having trouble making a reinforcement algorithm than can win the 2048 game. I have tried with deep Q (which I think is the simplest algorithm that should be able to learn a winning strategy). ...
32 views

### Why do we need recurrent neural networks instead of feed-forward neural networks? [duplicate]

Why do we need recurrent neural networks instead of feed-forward neural networks? What are the advantages of RNNs compared with FFNNs?
63 views

### Why is my neural network not able to approximate this function?

I'm trying to approximate the following function with a neural network (in Python). ...
195 views

### Why do we need convolutional neural networks instead of feed-forward neural networks?

Why do we need convolutional neural networks instead of feed-forward neural networks? What is the significance of a CNN? Even a feed-forward neural network will able to solve the image classification ...
68 views

### What are the most common feedforward neural networks?

What are the most common feedforward neural networks? What kind of inputs do they receive? For example, do they receive binary numbers, real numbers, vectors, or matrics? Is there such a taxonomy?
77 views

### How can feedforward neural networks act as contraction maps?

In graph neural networks, the Banach fixed-point theorem and Jacobi method it is described that the transition from one state to another be defined by a contraction map with a fixed-point. The autor ...
60 views

### Accuracy dropped when I ran the program the second time

I was following a tutorial about Feed Forward Networks and wrote this code for a simple FFN : ...
33 views

### Can neural networks always be assembled like Lego blocks?

BACKGROUND Consider a supervised problem which is based on two scalar features (1) and (2) as well as a third, "time-dependent", feature consisting of a sequence of five values (3)-(7). For ...
348 views

### What exactly is averaged when doing batch gradient descent?

I have a question about how the averaging works when doing mini-batch gradient descent. I think I now understood the general gradient descent algorithm, but only for online learning. When doing mini-...
62 views

### What are standard datasets for fully connected neural networks?

I am looking for datasets that are used as a testing standard in the fully connected neural networks (FCNN). For example, in the image recognition and CNN, CIFAR datasets are used in most of the ...
33 views

### How do we minimize loss for a single neuron with a feedback?

Suppose we had a series of single-dimensional data points $X = \{x_1, x_2, \dots, x_n \}$, where $n$ is the number of data points and there corresponding output values $T = \{t_1, t_2, \dots, t_n \}$. ...
26 views

### Can denoising auto-encoders be convolutional and fully connected?

I have been reading lately on autoencoders a lot. I just wanted to summarize my understanding of denoising autoencoders. As far as I understand they can be Fully connected (in which case, they will ...
70 views

### What happens if I train a network for more epochs, without using early stopping?

I have a question about training a neural network for more epochs even after the network has converged without using early stopping criterion. Consider the MNIST dataset and a LeNet 300-100-10 dense ...
71 views

### Why would my neural network run faster on my laptop than on my university's supercomputer?

I am trying to get my neural network running on my university's supercomputer in order to decrease its runtime (not for training, for testing - feedforward runs only). However, the Matlab function I ...
63 views

### Are there names for neural networks with a well-defined layer or neuron characteristics?

Are there names for neural networks with a well-defined layer or neuron characteristics? For example, a matrix that has the same number of rows and columns is called a square matrix. Is there an ...
26 views

### Neural network seems to just figure out the probability of a specific result

I am really new to neural networks, so i was following along with a video series, created by '3blue1brown' on youtube. I created an implementation of the network he explained in c++. I am attempting ...
81 views

### What is the name of this neural network architecture with layers that are also connected to non-neighbouring layers?

Consider a feedforward neural network. Suppose you have a layer of inputs, which is feedforward to a hidden layer, and feedforward both the input and hidden layers to an output layer. Is there a name ...
31 views

### How do I determine the best neural network architecture for a problem with 3 inputs and 12 outputs?

This post continues the topic in the following post: Is it possible to train a neural network with 3 inputs and 12 outputs?. I conducted several experiments in MATLAB and selected those neural ...
129 views

### How to make DNN learn multiplication/division?

A single neuron with 2 weights and identity activation can learn addition/subtraction as the 2 weights will converge to 1 and 1 (addition), or 1 and -1 (subtraction). However, for multiplication and ...
46 views

### Given an input $x \in R^{1\times d}$ and a network with $s$ hidden layers, is the time complexity of the forward pass $O(d^{2}s)$? [duplicate]

I have a neural network that takes as an input a vector of $x \in R^{1\times d}$ with $s$ hidden layers and each layer has $d$ neurons (including the output layer). If I understand correctly the ...