Questions tagged [feedforward-neural-networks]
For questions related to feedforward neural networks (FFNNs), which are also sometimes called multilayer perceptrons, but these two expressions may not always be interchangeable.
27
questions with no upvoted or accepted answers
3
votes
0
answers
615
views
How to use residual learning applied to fully connected networks?
Is there any reason why skip connections would not provide the same benefits to fully connected layers as it does for convolutional?
I've read the ResNet paper and it says that the applications should ...
3
votes
1
answer
119
views
Attempting to solve a optical character recognition task using a feed-forward network
I am doing some experimentation on neural networks, and for that I am trying to program a plain OCR task. I have learned CNNs are the best choice ,but for the time being and due to my inexperience, I ...
3
votes
0
answers
58
views
Using a "is_padding" attribute in your padding instead of simply zero vectors
Typical Feed Forward Neural Networks require a fixed sized input and output. So when you have variable sized input, it seems to be common practice to pad the input with zero vectors.
Why does it not ...
3
votes
1
answer
64
views
Which marketing-related classification challenges is a feed forward neural network suited to solve?
I am trying to think of some marketing-related classification challenges that a feed-forward neural network would be suited for.
Any ideas?
2
votes
1
answer
161
views
Closed networks vs Networks with a removed delay to predict new data
I've come across two types of neural networks to predict, both from Matlab, the closed structure and the net that removes one delay to find new data.
From Matlab's app generated scripts we see:
% ...
2
votes
0
answers
220
views
Bias gradient of layer before batch normalization always zero
From the original paper and this post we have that batch normalization backpropagation can be formulated as
I'm interested in the derivative of the previous layer outputs $x_i=\sigma(w X_i+b)$ with ...
2
votes
0
answers
289
views
Why should variance(output) equal variance(input) in Xavier Initialisation?
In a lot of explanations online for Xavier Initialization, I see the following:
With each passing layer, we want the variance to remain the same. This helps us keep the signal from exploding to a ...
2
votes
0
answers
43
views
How do I determine the best neural network architecture for a problem with 3 inputs and 12 outputs?
This post continues the topic in the following post:
Is it possible to train a neural network with 3 inputs and 12 outputs?.
I conducted several experiments in MATLAB and selected those neural ...
1
vote
0
answers
39
views
Is there a neural network architecture specialized for mapping lower-to-higher-dimensional data?
I am building a neural network that takes in a set of 86 parameters (primarily architecture-related, such as building floor area, kitchen size, number of a certain type of furniture, etc.) and outputs ...
1
vote
1
answer
83
views
How does a multidimensional vector get fed into a single node in a neural network?
I mostly develop neural networks completely from scratch, like without libraries. I've been seeing, especially in NLP tasks, entire vectors, often representing words, get fed into a single node. I'm ...
1
vote
0
answers
152
views
How can an MLP be implemented with convolutional layers?
I am studying the architecture of the network pointnet, specifically the MLPs stages of the pipeline highlighted in red in the following image (taken from the author page here):
It is strange to find ...
1
vote
0
answers
59
views
How to code mathematical functions in terms of neural networks?
I'm exploring how to solve differential equations using neural networks and discovered this Lagaris et. al. paper. In the paper, the solution to a differential equation is written as a mathematical ...
1
vote
0
answers
50
views
Can I help my neural network if I know the sign of the relationships between inputs and outputs
I am attempting to train a neural network where I can say the following:
For most inputs, I know the sign of the relationship between that input and several specific outputs. I.e. whatever set of ...
1
vote
0
answers
131
views
Does there exist functions for which the necessary number of nodes in a shallow neural network tends to infinity as approximation error tends to 0?
The Universal Approximation Theorem states (roughly) that any continuous function can be approximated to within an arbitrary precision $\varepsilon>0$ by a feedforward neural network with one ...
1
vote
0
answers
57
views
ReLU function converging to local optimum in one case and diverging in the other one
I implemented a simple neural network with 1 hidden layer. I used ReLU as activation function for the hidden layer and the output layer just uses the linear function.
To check my implementation I ...
1
vote
0
answers
38
views
What's the difference between RNNs and Feed Forward Neural Networks if a fixed size vector can preserve sequential information?
I was watching a Youtube video in which the problem of trying to predict the last word in a sentence was posed. The sentence was "I took my cat for a" and the last word was "walk"....
1
vote
0
answers
47
views
How do we minimize loss for a single neuron with a feedback?
Suppose we had a series of single-dimensional data points $X = \{x_1, x_2, \dots, x_n \}$, where $n$ is the number of data points and there corresponding output values $T = \{t_1, t_2, \dots, t_n \}$. ...
1
vote
0
answers
32
views
Can denoising auto-encoders be convolutional and fully connected?
I have been reading lately on autoencoders a lot. I just wanted to summarize my understanding of denoising autoencoders. As far as I understand they can be
Fully connected (in which case, they will ...
1
vote
0
answers
30
views
Neural network seems to just figure out the probability of a specific result
I am really new to neural networks, so i was following along with a video series, created by '3blue1brown' on youtube. I created an implementation of the network he explained in c++. I am attempting ...
1
vote
0
answers
173
views
Is this TensorFlow implementation of partial derivative of the cost with respect to the bias correct?
I have a neural network for MNIST classification which I am hard coding using TensorFlow 2.0. The neural network has an input layer consisting of 784 neurons (28 * 28), one hidden layer having "...
1
vote
0
answers
21
views
Self-organizing map using weighted non-euclidean distance to minimize variance of predictions
Let's say I have a dataset, each item/row of which has $\mathit{X + 1}$ characteristics where the last characteristic (i.e., the $\mathit{1}$) represents the some value I want to predict, $\mathit{Y}$,...
1
vote
0
answers
26
views
Matrix-output for FFNN?
Turns out that it looks like I will be approximating a 100x10 matrix in my project thesis. II have the following equation
$y = Dx$,
where $y$ is $(100 \times 1)$, $D$ is $100 \times 10$ and $x$ is $...
1
vote
2
answers
1k
views
How to create Partially Connected NNs with prespecified connections using Tensorflow?
I'd like to implement a partially connected neural network with ~3 to 4 hidden layers (a sparse deep neural network?) where I can specify which node connects to which node from the previous/next layer....
0
votes
0
answers
48
views
Why does my loss function fluctuate so much?
I have a loss function that I'm trying to maximise using a neural network.
While it does appear to increase and plateau over the training, it does so in a very "noisy" manner, spiking up and ...
0
votes
0
answers
12
views
How are groups created in maxout units when dividing the set of inputs 𝑧 into groups of 𝑘 values?
I don't get $G^(i)$the set of indices into the inputs for group $i$, $\{(i −1)k+ 1, . . . , ik\}$ when creating a maxout units/function, these thing that outputs the maximum element of groups: $$g(z)...
0
votes
0
answers
53
views
Backpropagation of position-wise feedforward neural network
I have read a paper entitled "Attention is all you need" by Vaswani et al. (2017). This paper use the so-called position-wise feedforward neural network, where the input of this network is a ...
0
votes
2
answers
788
views
Why isn't the loss of my neural network reduced after 2500 iterations?
I have developed a basic feedforward neural network from scratch to classify whether image is of cat or not cat. It works fine, but after 2500 iterations, my cost function is not reducing properly.
...