17 votes

How can I encode angle data to train neural networks?

The main problem with simply using the values $\alpha \in [0, 2\pi]$ is that semantically $0 = 2\pi$, but numerically $0$ and $2\pi$ are maximally far apart. A common way to encode this is by a vector ...
Chillston's user avatar
  • 1,674
13 votes
Accepted

Why are the initial weights of neural networks randomly initialised?

You shouldn't assign all to 0.5 because you'd have the "break symmetry" issue. http://www.deeplearningbook.org/contents/optimization.html Perhaps the only property known with complete ...
SmallChess's user avatar
  • 1,411
10 votes

How to train a neural network for a round based board game?

Great question! NN is very promising for this type of problem: Giraffe Chess. Lai's accomplishment was considered to be a pretty big deal, but unfortunately came just a few months before AlphaGo ...
DukeZhou's user avatar
  • 6,237
9 votes

Can LSTM neural networks be sped up by a GPU?

From Nvidia www (https://developer.nvidia.com/discover/lstm): Accelerating Long Short-Term Memory using GPUs The parallel processing capabilities of GPUs can accelerate the LSTM training and ...
pasaba por aqui's user avatar
9 votes
Accepted

Should we also shuffle the test dataset when training with SGD?

Short answer Shuffling affects learning (i.e. the updates of the parameters of the model), but, during testing or validation, you are not learning. So, it should not make any difference whether you ...
nbro's user avatar
  • 40.2k
8 votes
Accepted

What are the best known gradient-free training methods for deep learning?

There are several different algorithms that can be used for gradient free neural network training. Some of these algorithms include particle swarm optimization, genetic algorithms, simulated annealing,...
Aiden Grossman's user avatar
8 votes
Accepted

Is pooling a kind of dropout?

Dropout and Max-pooling are performed for different reasons. Dropout is a regularization technique, which affects only the training process (during evaluation, it is not active). The goal of dropout ...
Mark.F's user avatar
  • 446
8 votes

What could an oscillating training loss curve represent?

Overview As it has already been observed, your main problem, beside the training related issues like fixing the learning rate, is you have basically no chance to learn such a big model woth such a ...
Nicola Bernini's user avatar
8 votes
Accepted

What is the difference between training and testing in reinforcement learning?

What is reinforcement learning? In reinforcement learning (RL), you typically imagine that there's an agent that interacts, in time steps, with an environment by taking actions. On each time step $t$, ...
nbro's user avatar
  • 40.2k
8 votes
Accepted

Is it okay to use publicly available Instagram videos to train an AI?

Under US copyright law, this is probably fair use ...but beware of memorization. You may run into more trouble if the AI outputs things very similar to the original work. Also, consult a lawyer to ...
Ryan M's user avatar
  • 196
8 votes
Accepted

Deep Learning with Best-so-far instead of Where-you-are

Is this done in practice? Yes, this is done normally when using (lack of) improvements to validation metrics as a stop criterion, and many libraries support it as standard. Depending on the library, ...
Neil Slater's user avatar
  • 31.7k
8 votes

Does chatGPT learn or remember from (public) user input? Will it 'fess up to it? I could not get it to reveal

ChatGPT does not answer to just your last question, but to the whole dialog. It tries to continue the dialog in a way that has the same structure as the gigabytes of other texts it has studied. GPT-3 ...
Florian F's user avatar
  • 181
8 votes
Accepted

What makes reproducing a model like GPT3/GPT3.5/ChatGPT difficult?

Challenges to reproduce ChatGPT: Compute cost Collect training data Find the proper choice for network architecture + RL (OpenAI hasn't published all the details) Two interesting papers on training ...
Franck Dernoncourt's user avatar
7 votes

How to train a neural network for a round based board game?

I'm a chess player and my answer will be only on chess. Training a neural network with reinforcement learning isn't new, it has been done many times in the literature. I'll briefly explain the common ...
SmallChess's user avatar
  • 1,411
7 votes

Why are the initial weights of neural networks randomly initialised?

The initial weights in a neural network are initialized randomly because the gradient based methods commonly used to train neural networks do not work well when all of the weights are initialized to ...
Aiden Grossman's user avatar
7 votes

How do I choose the optimal batch size?

Here are a few guidelines, inspired by the deep learning specialization course, to choose the size of the mini-batch: If you have a small training set, use batch gradient descent (m < 200) In ...
Ayoub EL MAJJODI's user avatar
7 votes
Accepted

Is a GPU always faster than a CPU for training neural networks?

This changes according to your data and complexity of your models. See following article by microsoft. Their conclusion is The results suggest that the throughput from GPU clusters is always ...
Atilla Ozgur's user avatar
6 votes

What is the name of a human-inspired machine learning approach?

If it was based on how the human brain learns, it might have used hebbian learning. One example for such a network would be HTM.
BlindKungFuMaster's user avatar
6 votes
Accepted

What size of neural networks can be trained on current consumer grade GPUs? (1060,1070,1080)

Usually the problem is to fit the model into video RAM. If it does not, you cannot train your model at all without big efforts (like training parts of the model separately). If it does, time is your ...
C. Yduqoli's user avatar
6 votes
Accepted

Can LSTM neural networks be sped up by a GPU?

I found that there are cuDNN accelerated cells in Keras, for example, https://keras.io/layers/recurrent/#cudnnlstm. They are very fast. The normal LSTM cells are faster on CPU than on GPU.
Dieshe's user avatar
  • 289
6 votes

Why does training an SVM take so long? How can I speed it up?

The most likely explanation is that you're using too many training examples for your SVM implementation. SVMs are based around a kernel function. Most implementations explicitly store this as an NxN ...
John Doucette's user avatar
6 votes

How do I choose the optimal batch size?

From the blog A Gentle Introduction to Mini-Batch Gradient Descent and How to Configure Batch Size (2017) by Jason Brownlee. How to Configure Mini-Batch Gradient Descent Mini-batch gradient descent ...
6 votes
Accepted

What is the "thing" which is trained in AI model training

This answer applies to Machine Learning (ML) part of AI, as that seems to be what you are asking about. Please bear in mind that AI is still a broad church, including many other techniques than ML. ML,...
Neil Slater's user avatar
  • 31.7k
6 votes

What could an oscillating training loss curve represent?

Try lowering the learning rate. Such a loss curve can be indicative of a high learning rate. Due to a high learning rate the algorithm can take large steps in the direction of the gradient and miss ...
Sabyasachi Ghosh's user avatar
6 votes

Should I continue training if the neural network attains 100% training accuracy?

First of all, as mentioned by @Neil Slater in the comment - you need to have three splits into the train, validation and test set. One sometimes disregards the difference between validation and test ...
spiridon_the_sun_rotator's user avatar
6 votes

What is Lipschitz constraint and why it is enforced on discriminator?

The Lipschitz constraint is essentially that a function must have a maximum gradient. The specific maximum gradient is a hyperparameter. It's not mandatory for a discriminator to obey a Lipschitz ...
user253751's user avatar
6 votes

What is the reason we loop over epochs when training a neural network?

I am not expert on optimization, but I can share with you my knowledge of the topic. I think that the source of your confusion is that you assumed that, after the first epoch, we have reached a local ...
nbro's user avatar
  • 40.2k
5 votes
Accepted

How to shorten the development time of a neural network?

Your scenario is common. The most straightforward approach is to subsample your data randomly. Unless your data or your model has strong bias, your performance to the smaller data set should be ...
SmallChess's user avatar
  • 1,411
5 votes
Accepted

How can action recognition be achieved?

There are several approaches as to how this can be achieved. One recent study from 2015 about Action Recognition in Realistic Sports VideosPDF uses the action recognition framework based on the three ...
kenorb's user avatar
  • 10.5k

Only top scored, non community-wiki answers of a minimum length are eligible