12 votes
Accepted

Why are the initial weights of neural networks randomly initialised?

You shouldn't assign all to 0.5 because you'd have the "break symmetry" issue. http://www.deeplearningbook.org/contents/optimization.html Perhaps the only property known with complete ...
user avatar
  • 1,380
10 votes

How to train a neural network for a round based board game?

Great question! NN is very promising for this type of problem: Giraffe Chess. Lai's accomplishment was considered to be a pretty big deal, but unfortunately came just a few months before AlphaGo ...
user avatar
  • 6,067
9 votes

Can LSTM neural networks be sped up by a GPU?

From Nvidia www (https://developer.nvidia.com/discover/lstm): Accelerating Long Short-Term Memory using GPUs The parallel processing capabilities of GPUs can accelerate the LSTM training and ...
user avatar
8 votes
Accepted

Is it okay to use publicly available Instagram videos to train an AI?

Under US copyright law, this is probably fair use ...but beware of memorization. You may run into more trouble if the AI outputs things very similar to the original work. Also, consult a lawyer to ...
user avatar
  • 196
7 votes

How to train a neural network for a round based board game?

I'm a chess player and my answer will be only on chess. Training a neural network with reinforcement learning isn't new, it has been done many times in the literature. I'll briefly explain the common ...
user avatar
  • 1,380
7 votes

Why are the initial weights of neural networks randomly initialised?

The initial weights in a neural network are initialized randomly because the gradient based methods commonly used to train neural networks do not work well when all of the weights are initialized to ...
user avatar
7 votes
Accepted

Is pooling a kind of dropout?

Dropout and Max-pooling are performed for different reasons. Dropout is a regularization technique, which affects only the training process (during evaluation, it is not active). The goal of dropout ...
user avatar
  • 426
7 votes
Accepted

Is a GPU always faster than a CPU for training neural networks?

This changes according to your data and complexity of your models. See following article by microsoft. Their conclusion is The results suggest that the throughput from GPU clusters is always ...
user avatar
6 votes

What is the name of a human-inspired machine learning approach?

If it was based on how the human brain learns, it might have used hebbian learning. One example for such a network would be HTM.
user avatar
6 votes
Accepted

What are the best known gradient-free training methods for deep learning?

There are several different algorithms that can be used for gradient free neural network training. Some of these algorithms include particle swarm optimization, genetic algorithms, simulated annealing,...
user avatar
6 votes
Accepted

What size of neural networks can be trained on current consumer grade GPUs? (1060,1070,1080)

Usually the problem is to fit the model into video RAM. If it does not, you cannot train your model at all without big efforts (like training parts of the model separately). If it does, time is your ...
user avatar
6 votes

How do I choose the optimal batch size?

Here are a few guidelines, inspired by the deep learning specialization course, to choose the size of the mini-batch: If you have a small training set, use batch gradient descent (m < 200) In ...
user avatar
6 votes
Accepted

What is the "thing" which is trained in AI model training

This answer applies to Machine Learning (ML) part of AI, as that seems to be what you are asking about. Please bear in mind that AI is still a broad church, including many other techniques than ML. ML,...
user avatar
  • 23.8k
6 votes
Accepted

What is the difference between training and testing in reinforcement learning?

What is reinforcement learning? In reinforcement learning (RL), you typically imagine that there's an agent that interacts, in time steps, with an environment by taking actions. On each time step $t$, ...
user avatar
  • 33.8k
6 votes

Should I continue training if the neural network attains 100% training accuracy?

First of all, as mentioned by @Neil Slater in the comment - you need to have three splits into the train, validation and test set. One sometimes disregards the difference between validation and test ...
user avatar
6 votes

What is the reason we loop over epochs when training a neural network?

I am not expert on optimization, but I can share with you my knowledge of the topic. I think that the source of your confusion is that you assumed that, after the first epoch, we have reached a local ...
user avatar
  • 33.8k
5 votes
Accepted

How to shorten the development time of a neural network?

Your scenario is common. The most straightforward approach is to subsample your data randomly. Unless your data or your model has strong bias, your performance to the smaller data set should be ...
user avatar
  • 1,380
5 votes
Accepted

How can action recognition be achieved?

There are several approaches as to how this can be achieved. One recent study from 2015 about Action Recognition in Realistic Sports VideosPDF uses the action recognition framework based on the three ...
user avatar
  • 10k
5 votes

How can action recognition be achieved?

This study from 2012 uses 3D convolutional neural networks (CNN) for automated recognition of human actions in surveillance videos. The 3D CNN model extracts features from both the spatial and the ...
user avatar
  • 10k
5 votes
Accepted

Why is my test error lower than the training error?

You use dropout during traing to reduce overfitting, but this reduces the training accuracy. The dropout will not be used during testing, therefore the accuracy will be higher. That's normal behavior ...
user avatar
  • 1,644
5 votes
Accepted

How to refine K-means clustering on a data set?

The usual parameters to adjust in a k-means: Number of clusters (recall many clusters can have same label). Distance definition (euclidean is the most basic, Gauss is an improvement) Selection of ...
user avatar
5 votes
Accepted

Is random initialization of the weights the only choice to break the symmetry?

Randomising just b sort of works, but setting w to all zero causes severe problems with vanishing gradients, especially at the ...
user avatar
  • 23.8k
5 votes
Accepted

Can LSTM neural networks be sped up by a GPU?

I found that there are cuDNN accelerated cells in Keras, for example, https://keras.io/layers/recurrent/#cudnnlstm. They are very fast. The normal LSTM cells are faster on CPU than on GPU.
user avatar
  • 269
5 votes

Why does training an SVM take so long? How can I speed it up?

The most likely explanation is that you're using too many training examples for your SVM implementation. SVMs are based around a kernel function. Most implementations explicitly store this as an NxN ...
user avatar
5 votes

Which layer in a CNN consumes more training time: convolution layers or fully connected layers?

As CNN contains convolution operation, but DNN uses constructive divergence for training. CNN is more complex in terms of Big O notation. For reference: See Convolutional Neural Networks at ...
user avatar
5 votes

How do I choose the optimal batch size?

From the blog A Gentle Introduction to Mini-Batch Gradient Descent and How to Configure Batch Size (2017) by Jason Brownlee. How to Configure Mini-Batch Gradient Descent Mini-batch gradient descent ...
5 votes

Iteratively and adaptively increasing the network size during training

Neuroevolution Through Augmenting Topologies or NEAT may be what you are referring to. The original paper by Kenneth O. Stanley is here NEAT combines a neural network and a genetic algorithm. Instead ...
user avatar
  • 319
5 votes

What could an oscillating training loss curve represent?

Try lowering the learning rate. Such a loss curve can be indicative of a high learning rate. Due to a high learning rate the algorithm can take large steps in the direction of the gradient and miss ...
user avatar
5 votes

What could an oscillating training loss curve represent?

Overview As it has already been observed, your main problem, beside the training related issues like fixing the learning rate, is you have basically no chance to learn such a big model woth such a ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible