# Tag Info

### What is a recurrent neural network?

A recurrent neural network (RNN) is an artificial neural network that contains backward or self-connections, as opposed to just having forward connections, like in a feed-forward neural network (FFNN)....
• 35.6k
Accepted

### What is the time complexity of the forward pass algorithm of a feedforward neural network?

Let's suppose that we have an MLP with $15$ inputs, $20$ hidden neurons and $2$ output neurons. The operations performed are only in the hidden and output neurons, given that the input neurons only ...
• 35.6k

### What exactly is averaged when doing batch gradient descent?

Introduction First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't ...
• 35.6k
Accepted

### What is a recurrent neural network?

Recurrent neural networks (RNNs) are a class of artificial neural network architecture inspired by the cyclical connectivity of neurons in the brain. It uses iterative function loops to store ...
• 699
Accepted

### How to express a fully connected neural network succintly using linear algebra?

The equation $$\hat{y} = \sigma(xW_\color{green}{1})W_\color{blue}{2} \tag{1}\label{1}$$ is the equation of the forward pass of a single-hidden layer fully connected and feedforward neural network, i....
• 35.6k

### Why use a recurrent neural network over a feedforward neural network for sequence prediction?

An RNN or LSTM have the advantage of "remembering" the past inputs, to improve performance over prediction of a time-series data. If you use a neural network over like the past 500 characters, this ...
• 1,725
Accepted

### How do I decide the optimal number of layers for a neural network?

There is a technique called Pruning in neural networks, which is used just for this same purpose. The pruning is done on the number of hidden layers. The process ...
• 1,371
Accepted

### Can the hidden layer prior to the ouput layer have less hidden units than the output layer?

A layer with bigger number of nodes than previous one is something very common. Some examples are: strategies encoder-decoder (autoencoders) where the encoder typically has layers with a decreasing ...
• 1,272
Accepted

### Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?

You are talking about two different types of 'size'. The size of the input for a FFNN and a RNN must always remain fixed for the same network architecture, i.e. they take in a vector \$x \in \mathbb{R}^...
• 4,284
Accepted

### What is the significance of weights in a feedforward neural network?

You described a single-layer feedforward network. They can have multiple layers. The significance of the weights is that they make a linear transformation from the output of the previous layer and ...
• 381
Accepted

### Why do we need convolutional neural networks instead of feed-forward neural networks?

Why are CNNs useful? The main property of CNNs that make them more suitable than FFNNs to solve tasks where the inputs are images is that they perform convolutions (or cross-correlations). Convolution ...
• 35.6k

### How can the input order of pairs into a neural network not matter (i.e. symmetry)?

The problem you're describing is related to (if not a subset of) Shift Invariance. Shift invariance refers to any geometric translation of an input, but concatenation of a pair of tenors in 2 ...
• 4,583

### Why aren't there neural networks that connect the output of each layer to all next layers?

Actually, this already exists! I happened to make a presentation of a paper that talks about this topic. These networks are called DenseNets, which stands for densely connected convolutional networks....
• 41

### Why is the backpropagation algorithm used to train the multilayer perceptron?

According to wikipedia of backpropagation: In fitting a neural network, backpropagation computes the gradient of the loss function during supervised learning with respect to the weights of the ...
• 191

### Why would you implement the position-wise feed-forward network of the transformer with convolution layers?

I'm going to post another guess to this question - it won't be a complete answer, but hopefully it'll provide some direction towards finding a more legitimate answer. The feed-forward networks as ...
• 81

### Why would you implement the position-wise feed-forward network of the transformer with convolution layers?

1) The math is the exact same, so from an optimization or mathematical perspective there is no difference 2) Here are my guesses to a possible answer. Habit: People may just call one over the ...
• 2,329
Accepted

### Why use a recurrent neural network over a feedforward neural network for sequence prediction?

Assumptions Different model structures encode different assumptions - while we often make simplifying assumptions that aren't strictly correct, some assumptions are more wrong than others. For ...
• 853
Accepted

### How to make DNN learn multiplication/division?

In reallity any continous function on a compact can be approximated by a neural network having one hidden layer with a finite number of neurones (This is the Universal Approximation Theorem). Thus you ...
• 381
Accepted

### How do I design a neural network that breaks a 5-letter word into its corresponding syllables?

I would highly recommend modeling things differently with regard to how letters are presented to the model. While the problem is more natural, perhaps, for a Convolutional or Recurrent Neural Network, ...

### What exactly is averaged when doing batch gradient descent?

do I have to: forward propagate calculate error calculate all gradients ...repeatedly over all samples in the batch, and then average all gradients and apply the weight change? Yes, that is ...
• 25k
Accepted

### What's a good neural network for this problem?

A simple feed-forward neural network with at least one hidden layer would suffice in your problem, and can deal with arbitrary non-linear relationships between input and output. If you expect ...
• 25k

### How do I decide the optimal number of layers for a neural network?

You can take a look at bayesian hyperparameter optimization as a general method of optimizing loss (or anything) as a function of the hyperparameters. But note that in general the deeper your network ...
• 1,971

### Why does the fitness of my neural network to play tic-tac-toe keep oscillating?

What I notice is that the network's fitness keeps climbing up and falling down again. It seems that my current approach only evolves certain patterns on placing signs on the board and once random ...
• 9,604

### Neural network to detect "spam"?

First your questions: Yes, it is possible to accomplish this with a neural network, this is actually very similar to your current working model (the idea is the same, just different classes). So, ...
• 436
Accepted

### Feed forward neural network using numpy for IRIS dataset

You can not use the ready code directly without any manipulation. Because every piece of code is written for specific datasets. In the article that you mentioned, the writer created a small dataset ...
Accepted

### How to handle varying types and length of inputs in a feedforward neural network?

In short ANNs don't have problems with "different types" of data as long as they are represented using real numbers: the inputs for your ANN represent lengths and are easy to understand and ...
• 361