14
votes
Why is a bias parameter needed in neural networks?
It's not strictly "needed." In fact, if you look at things like Keras, you will see that layers have a use_bias parameter, which defaults to True, but ...
- 748
11
votes
Accepted
What is the time complexity of the forward pass algorithm of a feedforward neural network?
Let's suppose that we have an MLP with $15$ inputs, $20$ hidden neurons and $2$ output neurons. The operations performed are only in the hidden and output neurons, given that the input neurons only ...
- 37k
11
votes
What is a recurrent neural network?
A recurrent neural network (RNN) is an artificial neural network that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN)....
- 37k
11
votes
What exactly is averaged when doing batch gradient descent?
Introduction
First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't ...
- 37k
11
votes
Why is a bias parameter needed in neural networks?
Let's write some code, shall we? First I'll generate two 2D Gaussian blobs with means at (0,0) and at (3,3) and sigma = 1.0. The points for the blob at (0,0) will be in class ...
- 2,261
9
votes
Accepted
What is a recurrent neural network?
Recurrent neural networks (RNNs) are a class of artificial neural network
architecture inspired by the cyclical connectivity of neurons in the brain. It uses iterative function loops to store ...
- 699
8
votes
Accepted
How to express a fully connected neural network succintly using linear algebra?
The equation $$\hat{y} = \sigma(xW_\color{green}{1})W_\color{blue}{2} \tag{1}\label{1}$$ is the equation of the forward pass of a single-hidden layer fully connected and feedforward neural network, i....
- 37k
8
votes
Why is a bias parameter needed in neural networks?
No matter what you make $W_1$ and $W_2$, if $X_1$ is 0 and $X_2$ is 0 then $W_1X_1+W_2X_2$ is 0 which (in a typical classification application) means the classifier is completely unsure which class it ...
- 659
7
votes
Why use a recurrent neural network over a feedforward neural network for sequence prediction?
An RNN or LSTM have the advantage of "remembering" the past inputs, to improve performance over prediction of a time-series data. If you use a neural network over like the past 500 characters, this ...
- 1,725
7
votes
Accepted
How do I decide the optimal number of layers for a neural network?
There is a technique called Pruning in neural networks, which is used just for this same purpose.
The pruning is done on the number of hidden layers. The process ...
- 1,371
7
votes
Accepted
Can the hidden layer prior to the ouput layer have less hidden units than the output layer?
A layer with bigger number of nodes than previous one is something very common. Some examples are:
strategies encoder-decoder (autoencoders) where the encoder typically has layers with a decreasing ...
- 1,282
7
votes
Accepted
Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?
You are talking about two different types of 'size'. The size of the input for a FFNN and a RNN must always remain fixed for the same network architecture, i.e. they take in a vector $x \in \mathbb{R}^...
- 4,400
7
votes
Why is a bias parameter needed in neural networks?
If you have data generated from $y = 5\,x + 3$
How do you expect the simplest neural net $y = w_1 x$ to adjust the data ?
That is why $b$ is useful.
- 79
5
votes
Accepted
What is the significance of weights in a feedforward neural network?
You described a single-layer feedforward network. They can have multiple layers. The significance of the weights is that they make a linear transformation from the output of the previous layer and ...
- 381
5
votes
Why aren't there neural networks that connect the output of each layer to all next layers?
Actually, this already exists!
I happened to make a presentation of a paper that talks about this topic. These networks are called DenseNets, which stands for densely connected convolutional networks....
- 51
5
votes
Accepted
Why do we need convolutional neural networks instead of feed-forward neural networks?
Why are CNNs useful?
The main property of CNNs that make them more suitable than FFNNs to solve tasks where the inputs are images is that they perform convolutions (or cross-correlations).
Convolution
...
- 37k
5
votes
How can the input order of pairs into a neural network not matter (i.e. symmetry)?
The problem you're describing is related to (if not a subset of) Shift Invariance. Shift invariance refers to any geometric translation of an input, but concatenation of a pair of tenors in 2 ...
- 4,763
4
votes
What exactly is averaged when doing batch gradient descent?
do I have to:
forward propagate
calculate error
calculate all gradients
...repeatedly over all samples in the batch, and then average all gradients and apply the weight change?
Yes, that is ...
- 26.5k
4
votes
Accepted
What's a good neural network for this problem?
A simple feed-forward neural network with at least one hidden layer would suffice in your problem, and can deal with arbitrary non-linear relationships between input and output. If you expect ...
- 26.5k
4
votes
Why is the backpropagation algorithm used to train the multilayer perceptron?
According to wikipedia of backpropagation:
In fitting a neural network, backpropagation computes the gradient of the loss function during supervised learning with respect to the weights of the ...
- 191
3
votes
Why would you implement the position-wise feed-forward network of the transformer with convolution layers?
I'm going to post another guess to this question - it won't be a complete answer, but hopefully it'll provide some direction towards finding a more legitimate answer.
The feed-forward networks as ...
- 81
3
votes
Why would you implement the position-wise feed-forward network of the transformer with convolution layers?
1) The math is the exact same, so from an optimization or mathematical perspective there is no difference
2) Here are my guesses to a possible answer.
Habit: People may just call one over the ...
- 2,339
3
votes
Accepted
Why use a recurrent neural network over a feedforward neural network for sequence prediction?
Assumptions
Different model structures encode different assumptions - while we often make simplifying assumptions that aren't strictly correct, some assumptions are more wrong than others.
For ...
- 873
3
votes
Accepted
How to make DNN learn multiplication/division?
In reallity any continous function on a compact can be approximated by a neural network having one hidden layer with a finite number of neurones (This is the Universal Approximation Theorem). Thus you ...
- 381
3
votes
Neural network to detect "spam"?
The Project Summarized
The project goal appears to be a common one: Routing correspondence in an efficient manner to maintain good but low cost customer and public relations. A few features of the ...
- 7,375
3
votes
Accepted
How do I design a neural network that breaks a 5-letter word into its corresponding syllables?
I would highly recommend modeling things differently with regard to how letters are presented to the model. While the problem is more natural, perhaps, for a Convolutional or Recurrent Neural Network, ...
- 46
3
votes
Accuracy dropped when I ran the program the second time
It is common during the training of Neural Networks for accuracy to improve for a while and then get worse -- in general, This is caused by over-fitting. It's also fairly common for the Neural Network ...
- 1,004
2
votes
How do I decide the optimal number of layers for a neural network?
You can take a look at bayesian hyperparameter optimization as a general method of optimizing loss (or anything) as a function of the hyperparameters. But note that in general the deeper your network ...
- 1,991
2
votes
Compute Jacobian matrix of Deep learning model?
In the paper The Limitations of Deep Learning in Adversarial Settings, Papernot et. al., 2016, IEEE the chain rule is used, "To express $\nabla F(X∗)$ in terms of $X$ and constant values only."
...
- 7,375
2
votes
Why does the fitness of my neural network to play tic-tac-toe keep oscillating?
What I notice is that the network's fitness keeps climbing up and falling down again. It seems that my current approach only evolves certain patterns on placing signs on the board and once random ...
- 9,794
Only top scored, non community-wiki answers of a minimum length are eligible