Tag Info

8

Dropout means that every individual data point is only used to fit a random subset of the neurons. This is done to make the neural network more like an ensemble model. That is, just as a random forest is averaging together the results of many individual decision trees, you can see a neural network trained using dropout as averaging together the results of ...

7

Deep learning allows you to solve complex problems without necessarily being able to specify the important "features" or key input variables for the model in advance. To give an example, a problem that may be easily tackled without deep learning could be predicting the frequency and claim amounts of insurance vehicle claims, given historical claim data ...

6

The original paper1 that proposed neural network dropout is titled: Dropout: A simple way to prevent neural networks from overfitting. That tittle pretty much explains in one sentence what Dropout does. Dropout works by randomly selecting and removing neurons in a neural network during the training phase. Note that dropout is not applied during testing and ...

6

Usually the problem is to fit the model into video RAM. If it does not, you cannot train your model at all without big efforts (like training parts of the model separately). If it does, time is your only problem. But the difference in training time between consumer GPUs like the Nvidia 1080 and much more expensive GPU accelerators like the Nvidia K80 are not ...

5

Deep Learning these days mean a lot of things to a lot of people, its quickly becoming a buzz-word. But so far it still retains two very important conceptual properties: Does away with most feature engineering work. This was mentioned in the answer above, but this is very important. It really saves a lot of work. Allows you to make maximal use of ...

5

I can't speak for individual researchers, but I can guess why the community as a whole hasn't adopted this activation function. ReLU is just so incredibly cheap. This benefit continues to grow as networks grow deeper. Also, they work reasonably well. As pointed out in Searching for Activation Functions, the performance improvements of the other ...

5

Is there anything else I could do to improve accuracy for both training and testing? Yes, of course, there are a lot of methods if you want to try to improve your accuracy, some that I can mention: Try to use a more complex model: ResNet, DenseNet, etc. Try to use other optimizers: Adam, Adadelta, etc. Tune your hyperparameters (e.g. change your learning ...

3

When you want to compare Reinforcement Learning algorithms, you might want to compare the average rewards they generate and how fast and close they get to the optimal policy. However, in the case of comparing it to humans, you might want to compare the game results of all the games played. Reward Comparison Often Reinforcement Learning algorithms are ...

3

One option is not mentioned by malioboro is getting more data. Getting bigger dataset is almost always improve training results. If it's too hard to obtain more labeled data you can use data augmentation on existing data - small random transformations while keeping the same label. For images most common methods of augmentations are (applying padding if ...

3

As a caveat, I’d suggest that unless you’re pushing up against fundamental technological limits, computation speed and resources should be secondary to design rationale when developing a neural network architecture. That said, earlier this year I finished my MS thesis that involved bioinformatics analytics pipelines with whole genome sequencing data - that ...

3

There are some great answers here. The simplest explanation I can give for dropout is that it randomly excludes some neurons and their connections from the network, while training, to stop neurons from "co-adapting" too much. It has the effect of making each neuron apply more generally and is excellent for stopping overfitting for large neural networks.

3

Real time style transfer and neural doodle is very much possible and is an active topic I see users working on to improve upon. The basic idea is to do only feed forward propagation at test time and train with appropriate loss functions at train time. Perceptual Losses for Real-Time Style Transfer and Super-Resolution is a good starting point to understand ...

2

We are getting there, with as usual some trade-off between quality and speed. For example Lecture 8: Spatial Localization and Detection lecture shows some benchmarks (mAP = Mean Average Precision, higher is better; FPS = frame per second):

2

It is a labor-intensive process, but that does sound excessive. If you have a g2.8xlarge, make sure you are using the using the GPU flags for neural-style, which will cut your render time by an order of magnitude. That having been said, it is building a rather large network (depending on your parameters), and a 1024x768 image is a lot of input to work with....

2

I'll try to answer your questions using Geoffrey Hinton's ideas in dropout paper and his Coursera class. What purpose does the "dropout" method serve? Deep neural nets with a large number of parameters are very powerful machine learning systems. However, overfitting is a serious problem in such networks. Large networks are also slow to use, making ...

2

Deep learning allows you to not know the answer in order to ask the program a question. Their main benefit is their finite ability and flexible nature. The problem with procedural programing to solve problems is you have to know what the computer needs to do in order to solve the problem. What deep learning does is remove the requirement of the programmer ...

2

For most of the current use cases, where NNs are used in conjunction with images, the image quality (resolution, color depth) can be low. Consider image classification for example. The CNN extracts features from the image to tell different types of objects apart. Those features are pretty independent from the quality of the image (in reasonable bounds). ...

2

It depends on what you need. You can train any size of network on any resource. The problem is the time of training. If you want to train Inception on an average CPU it will take months to converge. So, it all depends on how long you can wait to see your results based on your network. As in neural nets we do not have only one operation but many (like ...

2

There are different ways to compare different kinds of AI techniques. As a starting point, be aware that "AI System" can mean an incredibly broad range of things. In popular culture, we usually think of a deployed system that uses AI techniques. These systems can only be compared on the basis of their performance, and their performance may have relatively ...

2

There is no easy rule for this. You can use transfer learning to select a model that works well on image classification. However the accuracy you achieve will be highly dependent on your training set. If your training set is "similar" in quantity and quality to what was used for the accuracy achieved by the transfer learning model in some application you ...

2

I'm not sure it's possible to help much because this is an experimental question. I'm afraid the only answer comes with testing many different options. I see a little thing that might be making your model a little worse, though: You're concatenating "relu" with "sigmoid". Placing two different nature values in the same array may make it more difficult ...

2

ReLU is piecewise linear function that outputs the received input directly if it's positive, or outputs a zero. i.e., $max(0, x)$ How significant is adding relu to full connected layers? ReLU, being an activation function, will determine what the output of the nodes in your FCs are. Since it's a non-linear function, one significance is it will allow the ...

2

Here is what I discovered empirically, trial and error. Since tuning the parameters are going to be environment specific, I'll lay out mine to give a better understanding of what I found to work for my case. Hopefully someone with better understanding of the algorithm will weigh in: Environment: A 2D map where an agent controls a simulated PC mouse pad and ...

2

First, I have not read and do not have that book. That said, I would interpret that statement in the context of the intractability of guaranteeing that the optimization function will find global minima in the loss surface. In other words, higher precision values will do nothing to improve whether we have descended into a global or local minimum. On the ...

1

It is common to have root mean squared error (RMSE) greater on the test dataset than on the training dataset (this is equal to having accuracy/score higher for model in training dataset than test dataset). This normally happens because the training data are assesed on the same data that have been learnt before, while the test dataset may have data that are ...

1

As you found $N$ is the number of nodes that are expanded. The cost of expansion of each node is equal to the number of children of that node. Hence, we use $b^*$ for each node. In other words, the total number of nodes that are involved in the expansion process is $N \times b^*$.

1

It is not clear to me that much will change. ARM makes many devices, mostly designed to consume as little power as possible. My guess is that most workstations will not contain ARM's ML processor, even if they contain an ARM CPU. ARM's ML processor can do machine learning. It is specifically optimized for training convolutional deep neural networks, and it ...

1

See here for a potential way to do it: http://infinity77.net/global_optimization/#motivation-motivation http://infinity77.net/global_optimization/#rules-the-rules You basically test the two (or more) optimization algorithms against known objective functions, with several random (but repeatable) starting points and then analyze the outcome.

Only top voted, non community-wiki answers of a minimum length are eligible