Questions tagged [optimizers]

For questions about optimization methods/algorithms (also know as optimizers) in the context of machine learning and other AI subfields. Examples of optimizers are plain (stochastic) gradient descent, Adam, SGD with momentum, Adagrad, and RMSprop.

Filter by
Sorted by
Tagged with
0 votes
2 answers
37 views

When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended?

When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended? I have an DNN training from an infinite stream of samples, that most likely won't ...
dronus's user avatar
  • 101
2 votes
1 answer
303 views

How do I use machine learning to create an optimization algorithm?

Let's say that I want to create an optimization algorithm, which is supposed to find an optimum value for a given objective function. Creating an optimization algorithm to explore through the search ...
sherl.lol's user avatar
1 vote
1 answer
2k views

What is uncentered variance and how it becomes equal to mean square in Adam?

I have been reading about Adam and AdamW (Here). The author mentioned that in "uncentered variance" we don't consider subtracting mean In this statement, the author is talking about ...
learner's user avatar
  • 151
2 votes
1 answer
281 views

Joined vs Separate optimizer for Actor-Critic

Say that I have a simple Actor-Critic architecture, (I am not familiar with Tensorflow, but) in Pytorch we need to specify the parameters when defining an optimizer (SGD, Adam, etc) and therefore we ...
Sanyou's user avatar
  • 165
1 vote
0 answers
325 views

Why does Adam optimizer work slower than Adagrad, Adadelta, and SGD for Neural Collaborative Filtering (NCF)?

I've been working on Neural Collaborative Filtering (NCF) recently to build a recommender system using Tensorflow Recommenders. Doing some hyperparameter tuning with different optimizers available in ...
bkaankuguoglu's user avatar
1 vote
1 answer
217 views

In the update rule of RMSprop, do we divide by a matrix?

I've been trying to understand RMSprop for a long time, but there's something that keeps eluding me. Here is a screenshot from this video by Andrew Ng. From the element-wise comment, from what I ...
Uriyasama's user avatar
2 votes
1 answer
233 views

What do we mean by "infrequent features"?

I am reading this blog post: https://ruder.io/optimizing-gradient-descent/index.html. In the section about AdaGrad, it says: It adapts the learning rate to the parameters, performing smaller updates (...
ava_punksmash's user avatar
2 votes
3 answers
16k views

What kind of optimizer is suggested to use for binary classification of similar images?

I have spent some time searching Google and wasn't able to find out what kind of optimization algorithm is best for binary classification when images are similar to one another. I'd like to read ...
bit_scientist's user avatar
9 votes
1 answer
1k views

What is the formula for the momentum and Adam optimisers?

In the gradient descent algorithm, the formula to update the weight $w$, which has $g$ as the partial gradient of the loss function with respect to it, is: $$w\ -= r \times g$$ where $r$ is the ...
Dan D.'s user avatar
  • 1,283