7

Dropout means that every individual data point is only used to fit a random subset of the neurons. This is done to make the neural network more like an ensemble model. That is, just as a random forest is averaging together the results of many individual decision trees, you can see a neural network trained using dropout as averaging together the results of ...


6

Deep learning allows you to solve complex problems without necessarily being able to specify the important "features" or key input variables for the model in advance. To give an example, a problem that may be easily tackled without deep learning could be predicting the frequency and claim amounts of insurance vehicle claims, given historical claim data ...


5

Is there anything else I could do to improve accuracy for both training and testing? Yes, of course, there are a lot of methods if you want to try to improve your accuracy, some that I can mention: Try to use a more complex model: ResNet, DenseNet, etc. Try to use other optimizers: Adam, Adadelta, etc. Tune your hyperparameters (e.g. change your learning ...


5

The original paper1 that proposed neural network dropout is titled: Dropout: A simple way to prevent neural networks from overfitting. That tittle pretty much explains in one sentence what Dropout does. Dropout works by randomly selecting and removing neurons in a neural network during the training phase. Note that dropout is not applied during testing and ...


5

Deep Learning these days mean a lot of things to a lot of people, its quickly becoming a buzz-word. But so far it still retains two very important conceptual properties: Does away with most feature engineering work. This was mentioned in the answer above, but this is very important. It really saves a lot of work. Allows you to make maximal use of ...


4

I can't speak for individual researchers, but I can guess why the community as a whole hasn't adopted this activation function. ReLU is just so incredibly cheap. This benefit continues to grow as networks grow deeper. Also, they work reasonably well. As pointed out in Searching for Activation Functions, the performance improvements of the other ...


3

One option is not mentioned by malioboro is getting more data. Getting bigger dataset is almost always improve training results. If it's too hard to obtain more labeled data you can use data augmentation on existing data - small random transformations while keeping the same label. For images most common methods of augmentations are (applying padding if ...


3

I think it's because the actual computation isn't being done by Python, but with the optimized libraries (like tensorflow) that are built with low level programming languages. The only parts that Python is being used for is the basic program structure and getting the data into the high performance machine learning libraries. Also Python is more like a ...


3

There are some great answers here. The simplest explanation I can give for dropout is that it randomly excludes some neurons and their connections from the network, while training, to stop neurons from "co-adapting" too much. It has the effect of making each neuron apply more generally and is excellent for stopping overfitting for large neural networks.


3

Real time style transfer and neural doodle is very much possible and is an active topic I see users working on to improve upon. The basic idea is to do only feed forward propagation at test time and train with appropriate loss functions at train time. Perceptual Losses for Real-Time Style Transfer and Super-Resolution is a good starting point to understand ...


2

We are getting there, with as usual some trade-off between quality and speed. For example Lecture 8: Spatial Localization and Detection lecture shows some benchmarks (mAP = Mean Average Precision, higher is better; FPS = frame per second):


2

It is a labor-intensive process, but that does sound excessive. If you have a g2.8xlarge, make sure you are using the using the GPU flags for neural-style, which will cut your render time by an order of magnitude. That having been said, it is building a rather large network (depending on your parameters), and a 1024x768 image is a lot of input to work with....


2

I'll try to answer your questions using Geoffrey Hinton's ideas in dropout paper and his Coursera class. What purpose does the "dropout" method serve? Deep neural nets with a large number of parameters are very powerful machine learning systems. However, overfitting is a serious problem in such networks. Large networks are also slow to use, making ...


2

Deep learning allows you to not know the answer in order to ask the program a question. Their main benefit is their finite ability and flexible nature. The problem with procedural programing to solve problems is you have to know what the computer needs to do in order to solve the problem. What deep learning does is remove the requirement of the programmer ...


2

For most of the current use cases, where NNs are used in conjunction with images, the image quality (resolution, color depth) can be low. Consider image classification for example. The CNN extracts features from the image to tell different types of objects apart. Those features are pretty independent from the quality of the image (in reasonable bounds). ...


2

There are different ways to compare different kinds of AI techniques. As a starting point, be aware that "AI System" can mean an incredibly broad range of things. In popular culture, we usually think of a deployed system that uses AI techniques. These systems can only be compared on the basis of their performance, and their performance may have relatively ...


2

Goodness is subjective. Reliable knowledge isn't possible with that flimsy a quality objective. The sturdy objective criteria you gave is 95%, so it is bad by that criteria. (I'm assuming that the 95% is expected for a given data set or a randomized sample from a given data set.) However, the 80% accuracy is good by the criteria where the you sum the ...


2

The graph seems to show the relative compute time of each component of the learning process, independent of the layer, using the typical meaning of layer in the context of artificial networks. The data is not unbelievable, in that one would expect a matrix-vector multiplication of an entire layer output would take more time than a matrix-vector ...


2

There is no easy rule for this. You can use transfer learning to select a model that works well on image classification. However the accuracy you achieve will be highly dependent on your training set. If your training set is "similar" in quantity and quality to what was used for the accuracy achieved by the transfer learning model in some application you ...


2

I'm not sure it's possible to help much because this is an experimental question. I'm afraid the only answer comes with testing many different options. I see a little thing that might be making your model a little worse, though: You're concatenating "relu" with "sigmoid". Placing two different nature values in the same array may make it more difficult ...


2

ReLU is piecewise linear function that outputs the received input directly if it's positive, or outputs a zero. i.e., $max(0, x)$ How significant is adding relu to full connected layers? ReLU, being an activation function, will determine what the output of the nodes in your FCs are. Since it's a non-linear function, one significance is it will allow the ...


1

It is common to have root mean squared error (RMSE) greater on the test dataset than on the training dataset (this is equal to having accuracy/score higher for model in training dataset than test dataset). This normally happens because the training data are assesed on the same data that have been learnt before, while the test dataset may have data that are ...


1

I think the best explanation is the Pareto Principle, where in this case, 80% of the the performance comes from 20% of the code. Most machine learning frameworks have a Python API that developers use, but the internals are usually implemented using C++ or CUDA (for GPU acceleration) or using specialized libraries like BLAS. This latter component is the 20% ...


1

You can expect that the inference time will strongly depend on particular hardware and software present on your platform. First, GPU equipped devices (eg NVidia TX) will outperform non-GPU equipped devices (eg. Intel Movidius). Second, software support (eg. cudnn, TensorRT) will make dramatic further impact. For instance, we have measured the inference ...


1

As you found $N$ is the number of nodes that are expanded. The cost of expansion of each node is equal to the number of children of that node. Hence, we use $b^*$ for each node. In other words, the total number of nodes that are involved in the expansion process is $N \times b^*$.


1

It is not clear to me that much will change. ARM makes many devices, mostly designed to consume as little power as possible. My guess is that most workstations will not contain ARM's ML processor, even if they contain an ARM CPU. ARM's ML processor can do machine learning. It is specifically optimized for training convolutional deep neural networks, and it ...


1

See here for a potential way to do it: http://infinity77.net/global_optimization/#motivation-motivation http://infinity77.net/global_optimization/#rules-the-rules You basically test the two (or more) optimization algorithms against known objective functions, with several random (but repeatable) starting points and then analyze the outcome.


1

Let me answer your question in two parts. If the Network is to be trained on images with high detail information(content); like, if you want to train a Network capable enough to pick and classify even smallest of the elements in that image. Eg- An image in a family picnic and you want to classify each fruit in the basket lying on the table, which would ...


1

Shallow layered networks are less capable of recursive or extended abstraction necessary for the kinds of generalization needed to handle complex tasks common in real world applications. It is the same problem as was discovered nearly a century ago in the analog world. One can try to reduce the components in the old tube radio design to lower its cost, but ...


Only top voted, non community-wiki answers of a minimum length are eligible