Questions tagged [feedforward-neural-networks]

For questions related to feedforward neural networks (FFNNs), which are also sometimes called multilayer perceptrons, but these two expressions may not always be interchangeable.

25 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3 votes
1 answer
103 views

Attempting to solve a optical character recognition task using a feed-forward network

I am doing some experimentation on neural networks, and for that I am trying to program a plain OCR task. I have learned CNNs are the best choice ,but for the time being and due to my inexperience, I ...
  • 51
3 votes
0 answers
54 views

Using a "is_padding" attribute in your padding instead of simply zero vectors

Typical Feed Forward Neural Networks require a fixed sized input and output. So when you have variable sized input, it seems to be common practice to pad the input with zero vectors. Why does it not ...
  • 153
3 votes
1 answer
61 views

Which marketing-related classification challenges is a feed forward neural network suited to solve?

I am trying to think of some marketing-related classification challenges that a feed-forward neural network would be suited for. Any ideas?
2 votes
1 answer
125 views

Closed networks vs Networks with a removed delay to predict new data

I've come across two types of neural networks to predict, both from Matlab, the closed structure and the net that removes one delay to find new data. From Matlab's app generated scripts we see: % ...
2 votes
0 answers
162 views

Bias gradient of layer before batch normalization always zero

From the original paper and this post we have that batch normalization backpropagation can be formulated as I'm interested in the derivative of the previous layer outputs $x_i=\sigma(w X_i+b)$ with ...
2 votes
0 answers
39 views

How do I determine the best neural network architecture for a problem with 3 inputs and 12 outputs?

This post continues the topic in the following post: Is it possible to train a neural network with 3 inputs and 12 outputs?. I conducted several experiments in MATLAB and selected those neural ...
  • 227
1 vote
0 answers
44 views

Can I help my neural network if I know the sign of the relationships between inputs and outputs

I am attempting to train a neural network where I can say the following: For most inputs, I know the sign of the relationship between that input and several specific outputs. I.e. whatever set of ...
  • 111
1 vote
0 answers
98 views

Does there exist functions for which the necessary number of nodes in a shallow neural network tends to infinity as approximation error tends to 0?

The Universal Approximation Theorem states (roughly) that any continuous function can be approximated to within an arbitrary precision $\varepsilon>0$ by a feedforward neural network with one ...
1 vote
0 answers
43 views

ReLU function converging to local optimum in one case and diverging in the other one

I implemented a simple neural network with 1 hidden layer. I used ReLU as activation function for the hidden layer and the output layer just uses the linear function. To check my implementation I ...
1 vote
1 answer
2k views

What is the space complexity for training a neural network using back-propagation?

Suppose that a simple feedforward neural network (FFNN) contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_l$ nodes in each layer. What is the space complexity to train this FFNN ...
1 vote
0 answers
176 views

Why should variance(output) equal variance(input) in Xavier Initialisation?

In a lot of explanations online for Xavier Initialization, I see the following: With each passing layer, we want the variance to remain the same. This helps us keep the signal from exploding to a ...
1 vote
0 answers
405 views

How to use residual learning applied to fully connected networks?

Is there any reason why skip connections would not provide the same benefits to fully connected layers as it does for convolutional? I've read the ResNet paper and it says that the applications should ...
1 vote
0 answers
30 views

What's the difference between RNNs and Feed Forward Neural Networks if a fixed size vector can preserve sequential information?

I was watching a Youtube video in which the problem of trying to predict the last word in a sentence was posed. The sentence was "I took my cat for a" and the last word was "walk"....
1 vote
0 answers
42 views

How do we minimize loss for a single neuron with a feedback?

Suppose we had a series of single-dimensional data points $X = \{x_1, x_2, \dots, x_n \}$, where $n$ is the number of data points and there corresponding output values $T = \{t_1, t_2, \dots, t_n \}$. ...
1 vote
0 answers
27 views

Can denoising auto-encoders be convolutional and fully connected?

I have been reading lately on autoencoders a lot. I just wanted to summarize my understanding of denoising autoencoders. As far as I understand they can be Fully connected (in which case, they will ...
  • 111
1 vote
0 answers
29 views

Neural network seems to just figure out the probability of a specific result

I am really new to neural networks, so i was following along with a video series, created by '3blue1brown' on youtube. I created an implementation of the network he explained in c++. I am attempting ...
1 vote
0 answers
154 views

Is this TensorFlow implementation of partial derivative of the cost with respect to the bias correct?

I have a neural network for MNIST classification which I am hard coding using TensorFlow 2.0. The neural network has an input layer consisting of 784 neurons (28 * 28), one hidden layer having "...
  • 205
1 vote
0 answers
21 views

Self-organizing map using weighted non-euclidean distance to minimize variance of predictions

Let's say I have a dataset, each item/row of which has $\mathit{X + 1}$ characteristics where the last characteristic (i.e., the $\mathit{1}$) represents the some value I want to predict, $\mathit{Y}$,...
1 vote
0 answers
25 views

Matrix-output for FFNN?

Turns out that it looks like I will be approximating a 100x10 matrix in my project thesis. II have the following equation $y = Dx$, where $y$ is $(100 \times 1)$, $D$ is $100 \times 10$ and $x$ is $...
1 vote
3 answers
1k views

How to create Partially Connected NNs with prespecified connections using Tensorflow?

I'd like to implement a partially connected neural network with ~3 to 4 hidden layers (a sparse deep neural network?) where I can specify which node connects to which node from the previous/next layer....
0 votes
0 answers
35 views

Combining loss function while training with another objective function

I would like to train a ReLU neural network minimizing an objective function that looks like this: $$L(W) + \eta + 1_{S}(\eta,W)$$ where $W$ is the set of weight matrices, $L(W)$ is a custom loss ...
  • 111
0 votes
0 answers
37 views

How can I do a time and space complexity comparison of a graph convolutional network vs. a feed forward network / MLP

How can I make a time and space complexity comparison of a graph convolutional network vs a feed-forward network / MLP? Does anybody have an idea how I can compare them?
0 votes
0 answers
22 views

Why would the training loss consistently increase over many epochs?

I am getting a very strange learning curve when I try training a neural network which I am not able to explain. I have never seen a learning curve that looks like this when it's a very simple and ...
  • 101
0 votes
0 answers
57 views

How to interpret Transformer output

In this (https://towardsdatascience.com/a-detailed-guide-to-pytorchs-nn-transformer-module-c80afbc9ffb1) article the author says, that the output of the ...
0 votes
2 answers
416 views

Why isn't the loss of my neural network reduced after 2500 iterations?

I have developed a basic feedforward neural network from scratch to classify whether image is of cat or not cat. It works fine, but after 2500 iterations, my cost function is not reducing properly. ...