Skip to main content
15 votes

Why is a bias parameter needed in neural networks?

It's not strictly "needed." In fact, if you look at things like Keras, you will see that layers have a use_bias parameter, which defaults to True, but ...
David Hoelzer's user avatar
12 votes

What exactly is averaged when doing batch gradient descent?

Introduction First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't ...
nbro's user avatar
  • 40.9k
11 votes
Accepted

What is the time complexity of the forward pass algorithm of a feedforward neural network?

Let's suppose that we have an MLP with $15$ inputs, $20$ hidden neurons and $2$ output neurons. The operations performed are only in the hidden and output neurons, given that the input neurons only ...
nbro's user avatar
  • 40.9k
11 votes

What is a recurrent neural network?

A recurrent neural network (RNN) is an artificial neural network that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN)....
nbro's user avatar
  • 40.9k
11 votes

Why is a bias parameter needed in neural networks?

Let's write some code, shall we? First I'll generate two 2D Gaussian blobs with means at (0,0) and at (3,3) and sigma = 1.0. The points for the blob at (0,0) will be in class ...
Kostya's user avatar
  • 2,534
9 votes
Accepted

What is a recurrent neural network?

Recurrent neural networks (RNNs) are a class of artificial neural network architecture inspired by the cyclical connectivity of neurons in the brain. It uses iterative function loops to store ...
naive's user avatar
  • 709
8 votes
Accepted

How to express a fully connected neural network succintly using linear algebra?

The equation $$\hat{y} = \sigma(xW_\color{green}{1})W_\color{blue}{2} \tag{1}\label{1}$$ is the equation of the forward pass of a single-hidden layer fully connected and feedforward neural network, i....
nbro's user avatar
  • 40.9k
8 votes

Why is a bias parameter needed in neural networks?

If you have data generated from $y = 5\,x + 3$ How do you expect the simplest neural net $y = w_1 x$ to adjust the data ? That is why $b$ is useful.
Mah Neh's user avatar
  • 89
8 votes

Why is a bias parameter needed in neural networks?

No matter what you make $W_1$ and $W_2$, if $X_1$ is 0 and $X_2$ is 0 then $W_1X_1+W_2X_2$ is 0 which (in a typical classification application) means the classifier is completely unsure which class it ...
Quitting Due To Antisemitism's user avatar
7 votes

Why use a recurrent neural network over a feedforward neural network for sequence prediction?

An RNN or LSTM have the advantage of "remembering" the past inputs, to improve performance over prediction of a time-series data. If you use a neural network over like the past 500 characters, this ...
Clement's user avatar
  • 1,745
7 votes
Accepted

How do I decide the optimal number of layers for a neural network?

There is a technique called Pruning in neural networks, which is used just for this same purpose. The pruning is done on the number of hidden layers. The process ...
Dawny33's user avatar
  • 1,371
7 votes
Accepted

Can the hidden layer prior to the ouput layer have less hidden units than the output layer?

A layer with bigger number of nodes than previous one is something very common. Some examples are: strategies encoder-decoder (autoencoders) where the encoder typically has layers with a decreasing ...
pasaba por aqui's user avatar
7 votes
Accepted

Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?

You are talking about two different types of 'size'. The size of the input for a FFNN and a RNN must always remain fixed for the same network architecture, i.e. they take in a vector $x \in \mathbb{R}^...
David's user avatar
  • 4,920
5 votes
Accepted

What is the significance of weights in a feedforward neural network?

You described a single-layer feedforward network. They can have multiple layers. The significance of the weights is that they make a linear transformation from the output of the previous layer and ...
Didami's user avatar
  • 391
5 votes

Why aren't there neural networks that connect the output of each layer to all next layers?

Actually, this already exists! I happened to make a presentation of a paper that talks about this topic. These networks are called DenseNets, which stands for densely connected convolutional networks....
Armando's user avatar
  • 51
5 votes
Accepted

Why do we need convolutional neural networks instead of feed-forward neural networks?

Why are CNNs useful? The main property of CNNs that make them more suitable than FFNNs to solve tasks where the inputs are images is that they perform convolutions (or cross-correlations). Convolution ...
nbro's user avatar
  • 40.9k
5 votes

How can the input order of pairs into a neural network not matter (i.e. symmetry)?

The problem you're describing is related to (if not a subset of) Shift Invariance. Shift invariance refers to any geometric translation of an input, but concatenation of a pair of tenors in 2 ...
Edoardo Guerriero's user avatar
4 votes

Why would you implement the position-wise feed-forward network of the transformer with convolution layers?

I'm going to post another guess to this question - it won't be a complete answer, but hopefully it'll provide some direction towards finding a more legitimate answer. The feed-forward networks as ...
mortonjt's user avatar
4 votes

What exactly is averaged when doing batch gradient descent?

do I have to: forward propagate calculate error calculate all gradients ...repeatedly over all samples in the batch, and then average all gradients and apply the weight change? Yes, that is ...
Neil Slater's user avatar
  • 32.7k
4 votes
Accepted

What's a good neural network for this problem?

A simple feed-forward neural network with at least one hidden layer would suffice in your problem, and can deal with arbitrary non-linear relationships between input and output. If you expect ...
Neil Slater's user avatar
  • 32.7k
4 votes

Why is the backpropagation algorithm used to train the multilayer perceptron?

According to wikipedia of backpropagation: In fitting a neural network, backpropagation computes the gradient of the loss function during supervised learning with respect to the weights of the ...
cinch's user avatar
  • 2,277
3 votes

Why would you implement the position-wise feed-forward network of the transformer with convolution layers?

1) The math is the exact same, so from an optimization or mathematical perspective there is no difference 2) Here are my guesses to a possible answer. Habit: People may just call one over the ...
mshlis's user avatar
  • 2,369
3 votes
Accepted

Why use a recurrent neural network over a feedforward neural network for sequence prediction?

Assumptions Different model structures encode different assumptions - while we often make simplifying assumptions that aren't strictly correct, some assumptions are more wrong than others. For ...
Peteris's user avatar
  • 883
3 votes
Accepted

How to make DNN learn multiplication/division?

In reallity any continous function on a compact can be approximated by a neural network having one hidden layer with a finite number of neurones (This is the Universal Approximation Theorem). Thus you ...
hola's user avatar
  • 381
3 votes

Neural network to detect "spam"?

The Project Summarized The project goal appears to be a common one: Routing correspondence in an efficient manner to maintain good but low cost customer and public relations. A few features of the ...
Douglas Daseeco's user avatar
3 votes
Accepted

How do I design a neural network that breaks a 5-letter word into its corresponding syllables?

I would highly recommend modeling things differently with regard to how letters are presented to the model. While the problem is more natural, perhaps, for a Convolutional or Recurrent Neural Network, ...
Gilad Tsur's user avatar
3 votes

Accuracy dropped when I ran the program the second time

It is common during the training of Neural Networks for accuracy to improve for a while and then get worse -- in general, This is caused by over-fitting. It's also fairly common for the Neural Network ...
Faizy's user avatar
  • 1,114
3 votes

Help with deep Q learning for 2048 game getting stuck

Q learning on its own isn't enough to learn a winning strategy for a game like 2048. 2048 requires predictive thinking for possible outcomes and good positional awareness. Performance of the agent is ...
Brale's user avatar
  • 2,406
2 votes

How do I decide the optimal number of layers for a neural network?

You can take a look at bayesian hyperparameter optimization as a general method of optimizing loss (or anything) as a function of the hyperparameters. But note that in general the deeper your network ...
k.c. sayz 'k.c sayz''s user avatar
2 votes

Compute Jacobian matrix of Deep learning model?

In the paper The Limitations of Deep Learning in Adversarial Settings, Papernot et. al., 2016, IEEE the chain rule is used, "To express $\nabla F(X∗)$ in terms of $X$ and constant values only." ...
Douglas Daseeco's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible