11 votes

What is a recurrent neural network?

A recurrent neural network (RNN) is an artificial neural network that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN)....
  • 35k
10 votes
Accepted

What is the time complexity of the forward pass algorithm of a feedforward neural network?

Let's suppose that we have an MLP with $15$ inputs, $20$ hidden neurons and $2$ output neurons. The operations performed are only in the hidden and output neurons, given that the input neurons only ...
  • 35k
10 votes

What exactly is averaged when doing batch gradient descent?

Introduction First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't ...
  • 35k
9 votes
Accepted

What is a recurrent neural network?

Recurrent neural networks (RNNs) are a class of artificial neural network architecture inspired by the cyclical connectivity of neurons in the brain. It uses iterative function loops to store ...
  • 699
8 votes
Accepted

How to express a fully connected neural network succintly using linear algebra?

The equation $$\hat{y} = \sigma(xW_\color{green}{1})W_\color{blue}{2} \tag{1}\label{1}$$ is the equation of the forward pass of a single-hidden layer fully connected and feedforward neural network, i....
  • 35k
7 votes

Why use a recurrent neural network over a feedforward neural network for sequence prediction?

An RNN or LSTM have the advantage of "remembering" the past inputs, to improve performance over prediction of a time-series data. If you use a neural network over like the past 500 characters, this ...
  • 1,715
7 votes
Accepted

How do I decide the optimal number of layers for a neural network?

There is a technique called Pruning in neural networks, which is used just for this same purpose. The pruning is done on the number of hidden layers. The process ...
  • 1,371
7 votes
Accepted

Can the hidden layer prior to the ouput layer have less hidden units than the output layer?

A layer with bigger number of nodes than previous one is something very common. Some examples are: strategies encoder-decoder (autoencoders) where the encoder typically has layers with a decreasing ...
7 votes
Accepted

Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?

You are talking about two different types of 'size'. The size of the input for a FFNN and a RNN must always remain fixed for the same network architecture, i.e. they take in a vector $x \in \mathbb{R}^...
5 votes
Accepted

What is the significance of weights in a feedforward neural network?

You described a single-layer feedforward network. They can have multiple layers. The significance of the weights is that they make a linear transformation from the output of the previous layer and ...
  • 381
5 votes
Accepted

Why do we need convolutional neural networks instead of feed-forward neural networks?

Why are CNNs useful? The main property of CNNs that make them more suitable than FFNNs to solve tasks where the inputs are images is that they perform convolutions (or cross-correlations). Convolution ...
  • 35k
5 votes

How can the input order of pairs into a neural network not matter (i.e. symmetry)?

The problem you're describing is related to (if not a subset of) Shift Invariance. Shift invariance refers to any geometric translation of an input, but concatenation of a pair of tenors in 2 ...
4 votes

Why aren't there neural networks that connect the output of each layer to all next layers?

Actually, this already exists! I happened to make a presentation of a paper that talks about this topic. These networks are called DenseNets, which stands for densely connected convolutional networks....
  • 41
4 votes

Accuracy dropped when I ran the program the second time

It is common during the training of Neural Networks for accuracy to improve for a while and then get worse -- in general, This is caused by over-fitting. It's also fairly common for the Neural Network ...
  • 625
4 votes

Why is the backpropagation algorithm used to train the multilayer perceptron?

According to wikipedia of backpropagation: In fitting a neural network, backpropagation computes the gradient of the loss function during supervised learning with respect to the weights of the ...
3 votes
Accepted

Why use a recurrent neural network over a feedforward neural network for sequence prediction?

Assumptions Different model structures encode different assumptions - while we often make simplifying assumptions that aren't strictly correct, some assumptions are more wrong than others. For ...
  • 853
3 votes
Accepted

How to make DNN learn multiplication/division?

In reallity any continous function on a compact can be approximated by a neural network having one hidden layer with a finite number of neurones (This is the Universal Approximation Theorem). Thus you ...
  • 381
3 votes

Why would you implement the position-wise feed-forward network of the transformer with convolution layers?

I'm going to post another guess to this question - it won't be a complete answer, but hopefully it'll provide some direction towards finding a more legitimate answer. The feed-forward networks as ...
3 votes
Accepted

How do I design a neural network that breaks a 5-letter word into its corresponding syllables?

I would highly recommend modeling things differently with regard to how letters are presented to the model. While the problem is more natural, perhaps, for a Convolutional or Recurrent Neural Network, ...
3 votes

What exactly is averaged when doing batch gradient descent?

do I have to: forward propagate calculate error calculate all gradients ...repeatedly over all samples in the batch, and then average all gradients and apply the weight change? Yes, that is ...
  • 24.7k
3 votes
Accepted

What's a good neural network for this problem?

A simple feed-forward neural network with at least one hidden layer would suffice in your problem, and can deal with arbitrary non-linear relationships between input and output. If you expect ...
  • 24.7k
2 votes

How do I decide the optimal number of layers for a neural network?

You can take a look at bayesian hyperparameter optimization as a general method of optimizing loss (or anything) as a function of the hyperparameters. But note that in general the deeper your network ...
2 votes

Why does the fitness of my neural network to play tic-tac-toe keep oscillating?

What I notice is that the network's fitness keeps climbing up and falling down again. It seems that my current approach only evolves certain patterns on placing signs on the board and once random ...
  • 9,519
2 votes

Neural network to detect "spam"?

First your questions: Yes, it is possible to accomplish this with a neural network, this is actually very similar to your current working model (the idea is the same, just different classes). So, ...
  • 426
2 votes
Accepted

Feed forward neural network using numpy for IRIS dataset

You can not use the ready code directly without any manipulation. Because every piece of code is written for specific datasets. In the article that you mentioned, the writer created a small dataset ...
2 votes
Accepted

How to handle varying types and length of inputs in a feedforward neural network?

In short ANNs don't have problems with "different types" of data as long as they are represented using real numbers: the inputs for your ANN represent lengths and are easy to understand and ...
2 votes

What is the name of this neural network architecture with layers that are also connected to non-neighbouring layers?

Such a network could be either a Residual Network or a Highway Network depending upon the underlying architecture of the skip layers. They are primarily used to to tackle the problem of vanishing ...
  • 360
2 votes
Accepted

What is the name of this neural network architecture with layers that are also connected to non-neighbouring layers?

This could be called a residual neural network (ResNet), which is a neural network with skip connections, that is, connections that skip layers. Here's a screenshot of a figure from the paper Deep ...
  • 35k

Only top scored, non community-wiki answers of a minimum length are eligible