49

Yes, indeed, neural networks are very prone to catastrophic forgetting (or interference). Currently, this problem is often ignored because neural networks are mainly trained offline (sometimes called batch training), where this problem does not often arise, and not online or incrementally, which is fundamental to the development of artificial general ...


28

Here's a snippet from an article by Gary Marcus In particular, they showed that standard deep learning nets often fall apart when confronted with common stimuli rotated in three dimensional space into unusual positions, like the top right corner of this figure, in which a schoolbus is mistaken for a snowplow: . . . Mistaking an ...


20

In theory, most neural networks can approximate any continuous function on compact subsets of $\mathbb{R}^n$, provided that the activation functions satisfy certain mild conditions. This is known as the universal approximation theorem (UAT), but that should not be called universal, given that there are a lot more discontinuous functions than continuous ones,...


17

Yes, the problem of forgetting older training examples is a characteristic of Neural Networks. I wouldn't call it a "flaw" though because it helps them be more adaptive and allows for interesting applications such as transfer learning (if a network remembered old training too well, fine tuning it to new data would be meaningless). In practice what you want ...


16

In our deep learning lecture, we discussed the following example (from Unmasking Clever Hans predictors and assessing what machines really learn (2019) by Lapuschkin et al.). Here the neural network learned a wrong way to identify a picture, i.E by identifying the wrong "relevant components". In the sensitivity maps next to the pictures, we can see that the ...


12

3D CNN's are used when you want to extract features in 3 Dimensions or establish a relationship between 3 dimensions. Essentially its the same as 2D convolutions but the kernel movement is now 3-Dimensional causing a better capture of dependencies within the 3 dimensions and a difference in output dimensions post convolution. The kernel on convolution ...


11

This is a very important problem that is usually overlooked. In fact, when training a neural network, there's often the implicit assumption that the data is independent and identically distributed, i.e., you do not expect the data to come from a distribution different than the distribution from which your training data comes, so there's also the implicit ...


11

Deeper models can have advantages (in certain cases) Most people will answer "yes" to your question, see e.g. Why are neural networks becoming deeper, but not wider? and Why do deep neural networks work well?. In fact, there are cases where deep neural networks have certain advantages compared to shallow ones. For example, see the following papers The ...


10

tl;dr I always like to think of Neural Networks as a generalization of logistic regression. I too don't like that, traditionally, when introducing Neural Networks, books start with biological neurons and synapses, etc. I think its more beneficial to start from statistics and linear regression, then logistic regression and then neural networks. A ...


9

There are several papers related to the topic, because there have been several attempts to show this from slightly different perspectives and using slightly different assumptions (e.g. assuming that certain activation functions are used). The article A visual proof that neural nets can compute any function (by Michael Nielsen) should give you some intuition ...


9

I agree that this is too broad, but here's a 1 sentence answer for most of them. The ones I left out (from the bottom of the chart) are very modern, and very specialized. I don't know much about them, so perhaps someone who does can improve this answer. Perceptron: Linear or logistic-like regression (and thus, classification). Feed Forward: Usually non-...


9

In a neural network (NN), a neuron can act as a linear operator, but it usually acts as a non-linear one. The usual equation of a neuron $i$ in layer $l$ of an NN is $$o_i^l = \sigma(\mathbf{x}_i^l \cdot \mathbf{w}_i^l + b_i^l),$$ where $\sigma$ is a so-called activation function, which is usually a non-linearity, but it can also be the identity ...


7

Let's suppose that we have an MLP with $15$ inputs, $20$ hidden neurons and $2$ output neurons. The operations performed are only in the hidden and output neurons, given that the input neurons only represent the inputs (so they do not perform any operation). Each hidden neuron performs a linear combination of its inputs followed by the application of a non-...


6

The auto-encoder (AE) can be used to learn a compressed representation (a vectorised hash value) of each observation in the training dataset, $z$, which can then be used to later retrieve the original (or similar) observation. The variational auto-encoder (VAE), a statistical variation of AE, can also be used to generate objects similar to the observations (...


6

All CNNs can be represented as vanilla networks on the flattened image data. Just to do so, you would need A LOT of parameters (most of which would be 0) for what CNNs do freely. You can think of a CNN as reusing a filter on a masked input (whichever receptive field it's looking at whatever point during the convolution) repetitively. In other words, fully ...


6

Since a neural network does iteratively learn its own weights I assume you mean the structure of the neural network - the number of layers and nodes per layer. If what I said above was your question, then yes, it most definitely is being explored. Even when a neural network is allowed to learn its own structure it still needs to be suited to a specific ...


6

If what you are asking is what is the intuition for using the derivative in backpropagation learning, instead of an in-depth mathematical explanation: Recall that the derivative tells you a function's sensitivity to change with respect to a change in its input. A high (absolute) value for the derivative at a certain point means that the function is very ...


6

I have an idea to find the optimal number of hidden neurons required in a neural network but I'm not sure how accurate it is. It's a complete non-starter, and there is a no such calculation possible in the general case (real-valued inputs to a neural network). Even with one input neuron it is not possible. That is because even with one input, the output ...


5

3D convolutions should when you want to extract spatial features from your input on three dimensions. For Computer Vision, they are typically used on volumetric images, which are 3D. Some examples are classifying 3D rendered images and medical image segmentation


5

Neuroevolution Through Augmenting Topologies or NEAT may be what you are referring to. The original paper by Kenneth O. Stanley is here NEAT combines a neural network and a genetic algorithm. Instead of using back propagation or gradient descent to "train" your network, NEAT creates a population of very simple neural networks (no connections) and evolves ...


5

This problem is called exploding gradients, resulting in an unstable network that at best cannot learn from the training data and at worst results in NaN weight values that can no longer be updated. One way to assure it is exploding gradients, is if loss is unstable and not improving, or if loss shows NaN value during training. Apart from the usual ...


5

Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e layer_output = Weights.layer_input + bias Consider a 2 layer neural network, the outputs from layer one will be: x2 = W1*x1 + b1 Now we pass the same input to the second layer, which will be x3 = W2x*2 + b2 Also x2 = W1*x1 + b1 Substituting ...


5

The loss function used is the triplet loss function. Let me explain it part by part. Notation The $f^a_i$ means the anchor input image. The $f^p_i$ means the postive input image, which corresponds to the same people as the anchor image. The $f^n_i$ corresponds to the negative sample, which is a different person(input image) then the anchor image. The ...


5

Dropout only ignores a portion of units during a single training batch update. Each training batch will use a different combination of units which gives them the best chance of that portion of them working together to generalize. Note the the weights for each unit are kept and will be updated during the next batch in which that unit is selected. During ...


5

It depends on the architecture of the neural network. However, in general, no, neurons at layer $l$ are not only affected by neurons at layer $l-1$. In the case of a multi-layer perceptron (or feed-forward neural network), only neurons at layer $l-1$ directly affect the neurons at layer $l$. However, neurons at layers $l-i$, for $i=2, \dots, l$, also ...


5

One of the things you may have missed out in your design is some arrow going from the last layer in the layer side back to the first layer. e.g. If you're thinking some thoughts you'll want those to keep going round and round in your head. At the moment while your design would have some way to learn and react to the environment, it wouldn't have any short ...


5

tl;dr: A batch size is the number of samples a network sees before updating its gradients. This number can range from a single sample to the whole training set. Empirically, there is a sweet spot in the range 1 to a few hundreds, where people experience the fastest training speeds. Check this article for more details. A more detailed explanation... If you ...


5

This is more in the direction of 'what kind of problems can be solved by neural networks'. In order to train a neural network you need a large set of training data which is labelled with correct/ incorrect for the question you are interested in. So for example 'identify all pictures that have a cat on them' is very suitable for neural networks. On the other ...


5

Yes, there are many, actually. A Google search turned this paper Artificial Neural Networks in Medical Diagnosis (2011) by Al-Shayea up. Not only are they used in disease diagnosis, but even with things like prescribing medicines. In fact, the top project for a hackathon at my school analysed thousands of research articles, and took a patient's medication ...


5

The difference between the validation and test set in my opinion should be explained in this way: the validation set is meant to be used multiple times. the test set is meant to be used only once. I think that the misunderstanding here arise because machine learning is mostly taught focusing only on a specific part of a large pipeline, which is the model ...


Only top voted, non community-wiki answers of a minimum length are eligible