New answers tagged

0

Yes, the optimal learning rate will differ for every change you make in the network. In fact finding the optimal learning rate is very computationally expensive, so you will normally only get a rough guess anyway. The learning rate is used to traverse an N dimensional loss landscape that changes drastically with even the smallest differences. If you add one ...


1

I personally can’t think of any good reason to apply data augmentation before splitting the dataset, though one may exist. The issue is that if you augment first and split later, you risk introducing unwanted correlations between your training and test datasets. In the paper you linked it sounds as if the training data is all procedurally derived from the ...


1

In my personal experience, that depends. Augmenting data for training purposes is valid, and can even improve performance, as you may be aware. For testing purposes, it may be valid. Let me give you two examples when that may be the case: Facial Recognition. Imagine that you have an augmentation function that can change the face pose (left/right pose, for ...


2

In the automatic differentiation procedure after backward pass the gradient with respect to the scalar is added to the current gradient. Without calling zero_grad you will have the sum of all gradients, calcluated before, with the current gradient. Therefore, optimizer.step() will do not this: w = w - eta * grad L[i] # L[i] - loss function for the i-th ...


6

First of all, as mentioned by @Neil Slater in the comment - you need to have three splits into the train, validation and test set. One sometimes disregards the difference between validation and test set. However they serve for different purposes. Here I would like to cite https://machinelearningmastery.com/difference-test-validation-datasets/ : Validation ...


4

Yes, you can fix (or freeze) some of the weights during the training of a neural network. In fact, this is done in the most common form of transfer learning (which is described here). I don't know exactly how this affects learning in general. In transfer learning, this is definitely beneficial, as we are freezing the weights that are associated with the ...


Top 50 recent answers are included