# Tag Info

12

You shouldn't assign all to 0.5 because you'd have the "break symmetry" issue. http://www.deeplearningbook.org/contents/optimization.html Perhaps the only property known with complete certainty is that the initial parameters need to “break symmetry” between different units. If two hidden units with the same activation function are connected to the ...

10

Great question! NN is very promising for this type of problem: Giraffe Chess. Lai's accomplishment was considered to be a pretty big deal, but unfortunately came just a few months before AlphaGo took the spotlight. (It all turned out well, in that Lai was subsequently hired by DeepMind, although not so well for the Giraffe engine;) I've found Lai's ...

9

From Nvidia www (https://developer.nvidia.com/discover/lstm): Accelerating Long Short-Term Memory using GPUs The parallel processing capabilities of GPUs can accelerate the LSTM training and inference processes. GPUs are the de-facto standard for LSTM usage and deliver a 6x speedup during training and 140x higher throughput during inference when compared to ...

8

Under US copyright law, this is probably fair use ...but beware of memorization. You may run into more trouble if the AI outputs things very similar to the original work. Also, consult a lawyer to help you apply the law to your specific situation. This is just information on general legal principles, not any specific situation, and also I'm not a lawyer. ...

7

I'm a chess player and my answer will be only on chess. Training a neural network with reinforcement learning isn't new, it has been done many times in the literature. I'll briefly explain the common strategies. The purpose of a network is to learn position evaluation. We all know a queen is stronger than a bishop, but can we make the network know about it ...

7

The initial weights in a neural network are initialized randomly because the gradient based methods commonly used to train neural networks do not work well when all of the weights are initialized to the same value. While not all of the methods to train neural networks are gradient based, most of them are, and it has been shown in several cases that ...

7

This changes according to your data and complexity of your models. See following article by microsoft. Their conclusion is The results suggest that the throughput from GPU clusters is always better than CPU throughput for all models and frameworks proving that GPU is the economical choice for inference of deep learning models. ... It is ...

6

If it was based on how the human brain learns, it might have used hebbian learning. One example for such a network would be HTM.

6

There are several different algorithms that can be used for gradient free neural network training. Some of these algorithms include particle swarm optimization, genetic algorithms, simulated annealing, and several others. Almost any optimization algorithm can be used to train a neural network. Here is an overview of some of the algorithms I listed: Particle ...

6

Usually the problem is to fit the model into video RAM. If it does not, you cannot train your model at all without big efforts (like training parts of the model separately). If it does, time is your only problem. But the difference in training time between consumer GPUs like the Nvidia 1080 and much more expensive GPU accelerators like the Nvidia K80 are not ...

6

Dropout and Max-pooling are performed for different reasons. Dropout is a regularization technique, which affects only the training process (during evaluation, it is not active). The goal of dropout is reduce unnecessary feature dependencies in the network, allowing it to be simpler and improves its generalization abilities (reduces overfitting). In simple ...

6

This answer applies to Machine Learning (ML) part of AI, as that seems to be what you are asking about. Please bear in mind that AI is still a broad church, including many other techniques than ML. ML, including neural networks for deep learning, and Reinforcement Learning (RL) is only a subset of AI - some AI techniques are more focused on the algorithm ...

6

First of all, as mentioned by @Neil Slater in the comment - you need to have three splits into the train, validation and test set. One sometimes disregards the difference between validation and test set. However they serve for different purposes. Here I would like to cite https://machinelearningmastery.com/difference-test-validation-datasets/ : Validation ...

5

Your scenario is common. The most straightforward approach is to subsample your data randomly. Unless your data or your model has strong bias, your performance to the smaller data set should be comparable. The accuracy might be lower, but the purpose is to do quick sanity check.

5

There are several approaches as to how this can be achieved. One recent study from 2015 about Action Recognition in Realistic Sports VideosPDF uses the action recognition framework based on the three main steps of feature extraction (shape, post or contextual information), dictionary learning to represent a video, and classification (BoW framework). A few ...

5

This study from 2012 uses 3D convolutional neural networks (CNN) for automated recognition of human actions in surveillance videos. The 3D CNN model extracts features from both the spatial and the temporal dimensions by performing 3D convolutions, thereby capturing the motion information encoded in multiple adjacent frames. A very similar deep learning ...

5

You use dropout during traing to reduce overfitting, but this reduces the training accuracy. The dropout will not be used during testing, therefore the accuracy will be higher. That's normal behavior if you work with dropout.

5

The usual parameters to adjust in a k-means: Number of clusters (recall many clusters can have same label). Distance definition (euclidean is the most basic, Gauss is an improvement) Selection of initial cluster positions. Data preprocessing (data normalization, ...)

5

Randomising just b sort of works, but setting w to all zero causes severe problems with vanishing gradients, especially at the start of learning. Using backpropagation, the gradient at the outputs of a layer L involves a sum multiplying the gradient of the inputs to layer L+1 by the weights (and not the biases) between the layers. This will be zero if the ...

5

I found that there are cuDNN accelerated cells in Keras, for example, https://keras.io/layers/recurrent/#cudnnlstm. They are very fast. The normal LSTM cells are faster on CPU than on GPU.

5

The most likely explanation is that you're using too many training examples for your SVM implementation. SVMs are based around a kernel function. Most implementations explicitly store this as an NxN matrix of distances between the training points to avoid computing entries over and over again. In your case, with 75% of 700,000 examples, this matrix will ...

5

As CNN contains convolution operation, but DNN uses constructive divergence for training. CNN is more complex in terms of Big O notation. For reference: See Convolutional Neural Networks at Constrained Time Cost for more details about the time complexity of CNNs See What is the time complexity of the forward pass algorithm of a neural network? and What is ...

5

Here are a few guidelines, inspired by the deep learning specialization course, to choose the size of the mini-batch: If you have a small training set, use batch gradient descent (m < 200) In practice: Batch mode: long iteration times Mini-batch mode: faster learning Stochastic mode: lose speed up from vectorization The typically mini-batch sizes ...

5

From the blog A Gentle Introduction to Mini-Batch Gradient Descent and How to Configure Batch Size (2017) by Jason Brownlee. How to Configure Mini-Batch Gradient Descent Mini-batch gradient descent is the recommended variant of gradient descent for most applications, especially in deep learning. Mini-batch sizes, commonly called “batch sizes” for brevity, ...

5

Neuroevolution Through Augmenting Topologies or NEAT may be what you are referring to. The original paper by Kenneth O. Stanley is here NEAT combines a neural network and a genetic algorithm. Instead of using back propagation or gradient descent to "train" your network, NEAT creates a population of very simple neural networks (no connections) and evolves ...

5

tl;dr: A batch size is the number of samples a network sees before updating its gradients. This number can range from a single sample to the whole training set. Empirically, there is a sweet spot in the range 1 to a few hundreds, where people experience the fastest training speeds. Check this article for more details. A more detailed explanation... If you ...

5

If the i.i.d (independent and identically distributed) assumption holds, shouldn't the training and validation trends be exactly the same? No, not necessarily. Let me explain why. If you assume your samples (aka examples, observations, data points, etc.) are i.i.d., this means that they come from the same distribution, e.g. a Gaussian $\mathcal{N}(0, 1)$ (...

5

What is reinforcement learning? In reinforcement learning (RL), you typically imagine that there's an agent that interacts, in time steps, with an environment by taking actions. On each time step $t$, the agent takes the action $a_t \in \mathcal{A}$ in the state $s_t \in \mathcal{S}$, receives a reward (or reinforcement) signal $r_t \in \mathbb{R}$ from the ...

4

Compare generated and real data All the results produced by G are always considered "wrong" by definition, even for a very good generator. You provide the discriminative neural network $D$ with a mix of results generated by the generator network $G$ and real results from an outside source, and then you train it to distinguish if the result was produced by ...

Only top voted, non community-wiki answers of a minimum length are eligible