Skip to main content
3 votes
Accepted

How to train a sample weight model for another ML model?

Sample Reweighting There's actually been a good amount of work on sample reweighting, namely learning a set of weights for each datapoint in your training data. However, the goal of these algorithms ...
Alexander Wan's user avatar
2 votes

When to use Pruning, Quantization , Distillation and others when optimizing speed

There are many survey papers on the topic of efficient deep learning. In general the decisions for pruning and quantisation are very dependant on your target hardware and downstream task. For example: ...
Roy's user avatar
  • 141
2 votes

Applicability of Holland's Schema Theorem to Genetic Algorithms with Non-Binary Individual Representations

It's been a while since I've been active in the research community, and it's possible that this is a controversial opinion, but I suspect it's still true that the Schema Theorem is not really used ...
deong's user avatar
  • 611
1 vote

Confusion about Adagrad/Rmsprop/Adam about the direction of change

as reported in the original paper: You can see that $v^t$ is a (corrected) exponential moving average of the second moment of the gradient This doesn't mean that the gradient is penalized if it has ...
Alberto's user avatar
  • 2,293
1 vote

Can you explain the Hinton's comment "Rprop is equivalent to using the gradient, but also dividing by the size of the gradient"?

"Rprop is equivalent to using the gradient" means Rprop fundamentally relies on information about the sign of the gradient of each weight to determine the direction of weight updates. Like ...
cinch's user avatar
  • 2,277
1 vote
Accepted

The SOTA of derivative-free optimization

The problem is not the input size but the model size. Indeed, derivative-free/zero-order optimization methods usually tend to estimate a descent direction that correlates with some notion of local ...
Alberto's user avatar
  • 2,293
1 vote

How to prevent update a pretrained model if a model is optimized with backpropagation?

You can freeze a model's parameters in pytorch using the following code: for param in encoder_model.parameters(): param.requires_grad = False or you can simply ...
Dennis Yang's user avatar
1 vote

Maximize a scoring function within the latent space of a generative model

I doubt that you can condition training on a scoring function, if such scoring is not used during training. Now, assuming your generator is differentiable, it's already been done the "n-step ...
Alberto's user avatar
  • 2,293
1 vote

Maximize a scoring function within the latent space of a generative model

For the first suggestion, are you suggesting doing gradient descent with objective $-S$? If so (and $S$ is differentiable), that's definitely possible. I would suggest looking at the PULSE paper. They ...
Alexander Wan's user avatar
1 vote
Accepted

Best way to generate fitness landscape when using higher dimensional data

I understood your question as being "I have a fitness function based on 8 parameters" how can I display that graphically. Visualising a multi-dimensional landscape is a hard problem. If you ...
Bruce Adams's user avatar
1 vote

When training a CNN, what are the hyperparameters to tune first?

It depends a lot on what type of architecture you are using. However, most of the standard architectures are quite stable and there is no need for much hypreparameter tuning. Choose whether you want ...
pi-tau's user avatar
  • 815

Only top scored, non community-wiki answers of a minimum length are eligible