A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

# Tag Info

55

First up, those images (even the first few) aren't complete trash despite being junk to humans; they're actually finely tuned with various advanced techniques, including another neural network. The deep neural network is the pre-trained network modeled on AlexNet provided by Caffe. To evolve images, both the directly encoded and indirectly encoded images, ...

45

As complexity rises, precise statements lose meaning and meaningful statements lose precision. ( Lofti Zadeh ). Fuzzy logic deals with reasoning that is approximate rather than fixed and exact. This may make the reasoning more meaningful for a human: Fuzzy logic is an extension of Boolean logic by Lotfi Zadeh in 1965 based on the mathematical theory of ...

28

The difference is mostly in the number of layers. For a long time, it was believed that "1-2 hidden layers are enough for most tasks" and it was impractical to use more than that, because training neural networks can be very computationally demanding. Nowadays, computers are capable of much more, so people have started to use networks with more layers and ...

25

The images that you provided may be unrecognizable for us. They are actually the images that we recognize but evolved using the Sferes evolutionary framework. While these images are almost impossible for humans to label with anything but abstract arts, the Deep Neural Network will label them to be familiar objects with 99.99% confidence. This result ...

19

Fuzzy logic is based on regular boolean logic. Boolean logic means you are working with truth values of either true or false (or 1 or 0 if you prefer). Fuzzy logic is the same apart from you can have truth values which are in-between true and false, that is to say you are working with any number between 0 (inclusive) and 1 (inclusive). The fact that you can ...

16

There is no direct way to find the optimal number of them: people empirically try and see (e.g., using cross-validation). The most common search techniques are random, manual, and grid searches. There exist more advanced techniques such as Gaussian processes, e.g. Optimizing Neural Network Hyperparameters with Gaussian Processes for Dialog Act ...

13

All answers here are great, but, for some reason, nothing has been said so far on why this effect should not surprise you. I'll fill the blank. Let me start with one requirement that is absolutely essential for this to work: the attacker must know neural network architecture (number of layers, size of each layer, etc). Moreover, in all cases that I examined ...

11

Formally, a single hidden layer is sufficient to approximate a continuous function to any desired degree of accuracy, so in that sense, you never need more than 1. This is called the Universal Approximation Theorem. Finding the best topology for a given problem is an open research problem. As far as I know, there are few universal 'rules of thumb' for this. ...

10

Convolutional Nets (CNN) rely on mathematical convolution (e.g. 2D or 3D convolutions), which is commonly used for signal processing. Images are a type of signal, and convolution can equally be used on sound, vibrations, etc. So, in principle, CNNs can find applications to any signal, and probably more. In practice, there exists already work on NLP (as ...

10

A deep neural network (DNN) is nothing but a neural network which has multiple layers, where multiple can be subjective. IMHO, any network which has 6 or 7 or more layers is considered deep. So, the above would form a very basic definition of a deep network.

10

An important question that does not yet have a satisfactory answer in neural network research is how DNNs come up with the predictions they offer. DNNs effectively work (though not exactly) by matching patches in the images to a "dictionary" of patches, one stored in each neuron (see the youtube cat paper). Thus, it may not have a high level view of the ...

9

A deep neural network is just a (feed-forward) neural network with many layers. However, deep belief networks, Deep Boltzman networks, etc., are not considered (debatable) deep neural networks, as their topology is different (they ave undirected networks in their topology). See also this: https://stats.stackexchange.com/a/59854/84191.

7

It's analogous to analogue versus digital, or the many shades of gray in between black and white: when evaluating the truthiness of a result, in binary boolean it's either true or false (0 or 1), but when utilizing fuzzy logic, it's an estimated probability between 0 and 1 (such as 0.75 being mostly probably true). It's useful for making calculated decisions ...

7

For a more intelligent approach than random or exhaustive searches, you could try a genetic algorithm such as NEAT http://nn.cs.utexas.edu/?neat. However, this has no guarantee to find a global optima, it is simply an optimization algorithm based on performance and is therefore vulnerable to getting stuck in a local optima.

7

Dropout means that every individual data point is only used to fit a random subset of the neurons. This is done to make the neural network more like an ensemble model. That is, just as a random forest is averaging together the results of many individual decision trees, you can see a neural network trained using dropout as averaging together the results of ...

7

The neural network is typically a set size once it's created. You'd have to create a network big enough for your data-set.

7

Everyone dealing with neural networks misses an important point when comparing systems with human like intelligence. A human takes many months to do anything intelligible, let alone being able to solve problems where adult humans can barely manage. That and the size of human brain is enormous compared to our neural networks. Direction might be right, but the ...

7

The white noise that fools DNNs isn't really white noise. It has been altered in the same way as the synthetic misclassified pictures have been altered. You have to change many input pixels in exactly such a way, that these little changes aren't perceptible, but propagated through the network add up to a misclassification. This is not going to happen by ...

7

There is stuff like the Universal Approximation Theorem. There are also investigations into the loss surface of neural networks. And classics like this explanation of the vanishing gradient problem. But I'm afraid the mathematical theory of neural networks only exists in bits and pieces in many different papers. And many of the most important questions ...

6

Generalization error is the error obtained by applying a model to data it has not seen before. So, if you want to measure generalization error, you need to remove a subset from your data and don't train your model on it. After training, you verify your model accuracy (or other performance measures) on the subset you have removed since your model hasn't seen ...

6

The first thing is to define what is a «good» and a «bad» sound. This is an extremely tricky issue, since the networks need numeric inputs. And music is whole bunch of numbers. I know from people doing research in identifying how similar two sounds are, and imitation, say: you hear a sound and try to make another that sounds like it. Like when you hum a ...

6

Have you read the book The Master Algorithm: by Pedro Domingos? He discusses the present day machine learning algorithms... Their strengths, weaknesses and applications... Deep Neural Network Genetic Algorithm Bayesian Network Support Vector Machine Inverse Deduction

6

Deep Learning is mostly successful in supervised learning, whereas the brain builds categories mostly in an unsupervised way. We don't yet know how to do that. (Take a look at google brain: 16,000 cores and all this thing can do is recognise cats and human faces with pretty abysmal accuracy.) Deep Learning uses highly unstructured activations, i.e. the high ...

5

How is it possible that deep neural networks are so easily fooled? Deep neural networks are easily fooled by giving high confidence predictions for unrecognizable images. How is this possible? Can you please explain ideally in plain English? Intuitively, extra hidden layers ought to make the network able to learn more complex classification functions, ...

5

The simple answer is "no, they aren't limited to images": CNNs are also being used for natural language processing. (See here for an introduction.) I haven't seen them applied to graphical data yet, but I haven't looked; there are some obvious things to try and so I'm optimistic that it would work.

5

Usually you keep track of training loss and validation loss and apply proper regularization technique (L1, L2, dropout, dropconnect, ...). The more interesting technique is to observe your validation loss with respect to the number of parameters in the network (often controlled by the number of layers/feature maps). If the validation starts dropping with ...

5

Deep networks have two main differences with 'normal' networks. The first is that computational power and training datasets have grown immensely, meaning that it's practical to run larger networks and statistically valid (that is, we have enough training examples that we won't just run into over-fitting problems with larger networks). The second is that ...

5

Generally researchers (Ghandar et al, Michalewicz, Lam) have used the profit or return on investment (ROI) as a reward (fitness) function. $ROI = \frac{ \left[\sum_{t=1}^T (Price_t - sc) \times I_s(t) \right] - \left[ \sum_{t=1}^T (Price_t + bc) \times I_b(t) \right] }{ \left[ \sum_{t=1}^T (Price_t + bc) \times I_b(t) \right] }$ where $I_b(t)$ and $I_s(t)$ ...

5

Neuroevolution Through Augmenting Topologies or NEAT may be what you are referring to. The original paper by Kenneth O. Stanley is here NEAT combines a neural network and a genetic algorithm. Instead of using back propagation or gradient descent to "train" your network, NEAT creates a population of very simple neural networks (no connections) and evolves ...

4

The original paper1 that proposed neural network dropout is titled: Dropout: A simple way to prevent neural networks from overfitting. That tittle pretty much explains in one sentence what Dropout does. Dropout works by randomly selecting and removing neurons in a neural network during the training phase. Note that dropout is not applied during testing and ...

Only top voted, non community-wiki answers of a minimum length are eligible