28

Here's a snippet from an article by Gary Marcus In particular, they showed that standard deep learning nets often fall apart when confronted with common stimuli rotated in three dimensional space into unusual positions, like the top right corner of this figure, in which a schoolbus is mistaken for a snowplow: . . . Mistaking an ...


19

In theory, most neural networks can approximate any continuous function on compact subsets of $\mathbb{R}^n$, provided that the activation functions satisfy certain mild conditions. This is known as the universal approximation theorem (UAT), but that should not be called universal, given that there are a lot more discontinuous functions than continuous ones,...


15

In our deep learning lecture, we discussed the following example (from Unmasking Clever Hans predictors and assessing what machines really learn (2019) by Lapuschkin et al.). Here the neural network learned a wrong way to identify a picture, i.E by identifying the wrong "relevant components". In the sensitivity maps next to the pictures, we can see that the ...


12

Deeper models can have advantages (in certain cases) Most people will answer "yes" to your question, see e.g. Why are neural networks becoming deeper, but not wider? and Why do deep neural networks work well?. In fact, there are cases where deep neural networks have certain advantages compared to shallow ones. For example, see the following papers The ...


11

This is a very important problem that is usually overlooked. In fact, when training a neural network, there's often the implicit assumption that the data is independent and identically distributed, i.e., you do not expect the data to come from a distribution different than the distribution from which your training data comes, so there's also the implicit ...


6

Before anything, the function you have wrote for the network lacks the bias variables (I'm sure you used bias to get those beautiful images, otherwise your tanh network had to start from zero). Generally I would say it's impossible to have a good approximation of sinus with just 3 neurons, but if you want to consider one period of sinus, then you can do ...


6

A layer with bigger number of nodes than previous one is something very common. Some examples are: strategies encoder-decoder (autoencoders) where the encoder typically has layers with a decreasing number of nodes (until the compressed/encoded data) and the decoder has layers increasing in number of nodes. bidirectional recurrent networks where in the ...


5

Dropout only ignores a portion of units during a single training batch update. Each training batch will use a different combination of units which gives them the best chance of that portion of them working together to generalize. Note the the weights for each unit are kept and will be updated during the next batch in which that unit is selected. During ...


5

There is a recent development in research that was looking into effectiveness of neural networks on arithmetic. Interestingly, feed-forward neural networks (MLPs) with various activation functions as well as LSTMs (RNNs which are Turing-complete) are not able to model simple arithmetic operations (e.g. addition/multiplication), they designed a new logic unit ...


5

It depends on the architecture of the neural network. However, in general, no, neurons at layer $l$ are not only affected by neurons at layer $l-1$. In the case of a multi-layer perceptron (or feed-forward neural network), only neurons at layer $l-1$ directly affect the neurons at layer $l$. However, neurons at layers $l-i$, for $i=2, \dots, l$, also ...


5

One of the things you may have missed out in your design is some arrow going from the last layer in the layer side back to the first layer. e.g. If you're thinking some thoughts you'll want those to keep going round and round in your head. At the moment while your design would have some way to learn and react to the environment, it wouldn't have any short ...


5

There is no strict definition of suitability of an activation function for neural networks. Instead there are a number of desirable traits, and functions that don't meet them or come close enough may perform badly in general (but those functions may still work in specific cases) If you are using gradient descent as a training method, then differentiability ...


5

The diagram you show works at least partially for describing both individual neurons and layers of those neurons. However, the "incoming" data lines on the left represent all inputs under consideration, typically a vector of all inputs to the cell. That includes all data from current time steps (from input layer or earlier LSTM or time-distributed layers) - ...


5

"Assuming that we have sufficient data..." — that's quite a big assumption. Also, traditional methods are well understood, while neural networks (and especially deep learning) is still something of a black box: you train it, and then you get a mapping from input to output. But you don't really know how that mapping is achieved. It's not only about ...


5

tl;dr: A batch size is the number of samples a network sees before updating its gradients. This number can range from a single sample to the whole training set. Empirically, there is a sweet spot in the range 1 to a few hundreds, where people experience the fastest training speeds. Check this article for more details. A more detailed explanation... If you ...


5

This is more in the direction of 'what kind of problems can be solved by neural networks'. In order to train a neural network you need a large set of training data which is labelled with correct/ incorrect for the question you are interested in. So for example 'identify all pictures that have a cat on them' is very suitable for neural networks. On the other ...


5

Yes, there are many, actually. A Google search turned this paper Artificial Neural Networks in Medical Diagnosis (2011) by Al-Shayea up. Not only are they used in disease diagnosis, but even with things like prescribing medicines. In fact, the top project for a hackathon at my school analysed thousands of research articles, and took a patient's medication ...


5

The difference between the validation and test set in my opinion should be explained in this way: the validation set is meant to be used multiple times. the test set is meant to be used only once. I think that the misunderstanding here arise because machine learning is mostly taught focusing only on a specific part of a large pipeline, which is the model ...


5

Pooling has multiple benefits Robust feature detection. Makes it computationally feasible to have deeper CNNs Robust Feature Detection Think of max-pooling (most popular) for understanding this. Consider a 2*2 box/unit in one layer which is mapped to only 1 box/unit in the next layer (Basically pooling). Let's say the feature map (kernel) detects a petal ...


4

There is nothing stopping you, you can setup Dense Neural Networks to have any size inputs or outputs (simple proof is to imagine a single layer NN with no activation is just a linear transform and given input dim $n$ and output dim $m$, it's just a matrix of $n$ x $m$, trivially this works though with any number of hidden layers) The better question is ...


4

Let's first recapitulate why the function that calculates the maximum between two or more numbers, $z=\operatorname{max}(x_1, x_2)$, is not a linear function. A linear function is defined as $y=f(x) = ax + b$, so $y$ linearly increases with $x$. Visually, $f$ corresponds to a straight line (or hyperplane, in the case of 2 or more input variables). If $z$ ...


4

I don't know if it might be of use, but many areas of NLP are still hard to tackle, and even if deep models achieve the state of the art results, they usually beat baseline shallow models by very few percentage points. One example that I've had the opportunity to work on is stance classification 1. In many datasets, the best F score achievable is around 70%....


4

What are the parameters in a convolutional layer? The (learnable) parameters of a convolutional layer are the elements of the kernels (or filters) and biases (if you decide to have them). There are 1d, 2d and 3d convolutions. The most common are 2d convolutions, which are the ones people usually refer to, so I will mainly focus on this case. 2d ...


4

As far as I know, the sigmoid is often used as the activation function of the output layer mainly because it is a convenient way of producing an output $p \in [0, 1]$, which can be interpreted as a probability, although that can be misleading or even wrong (if you interpret it as an uncertainty too). You may require the output of the neural network to be a ...


4

If the i.i.d (independent and identically distributed) assumption holds, shouldn't the training and validation trends be exactly the same? No, not necessarily. Let me explain why. If you assume your samples (aka examples, observations, data points, etc.) are i.i.d., this means that they come from the same distribution, e.g. a Gaussian $\mathcal{N}(0, 1)$ (...


4

There are actually quite a few. Personally I would say these courses have high quality and strong focus on practice: Standford computer vision cs231. Check the assignments materials on this page. This course has good explanation/exercises of how generally neural nets and backprop works. Fastai course notebooks. You can listen to the lectures as well, but ...


4

The book Grokking Deep Learning, by Andrew Trask (a PhD student at Oxford University and a research scientist at DeepMind), a wonderful, clean, and plain-English discussion of the basic mechanics that go on under the hood of neural networks - from data flow to updating of weights. It is written without a slant on normally-wonky math, the concepts are ...


4

Why are CNNs useful? The main property of CNNs that make them more suitable than FFNNs to solve tasks where the inputs are images is that they perform convolutions (or cross-correlations). Convolution The convolution is an operation (more precisely, a linear operator) that takes two functions $f$ and $h$ and produces another function $g$. It's often ...


4

It is definitley possible to make the links between neurons use more complex functions. Provided those functions are differentiable, backpropagation still works, and the resulting compound function might be able to learn something useful. The general name for such a thing is a computational graph and the standardised structures used in most neural networks ...


4

Let's quickly get out our copies of Deep Learning by Goodfellow et al. (2016). More specifically, I'm referring to page 276. On this page, the authors argue for a relatively small minibatch size, since there are less than linear returns for estimating the gradient when increasing the minibatch size. Returns here refer to the reduction of the standard error ...


Only top voted, non community-wiki answers of a minimum length are eligible