Questions tagged [optimizers]

For questions about optimization methods/algorithms (also know as optimizers) in the context of machine learning and other AI subfields. Examples of optimizers are plain (stochastic) gradient descent, Adam, SGD with momentum, Adagrad, and RMSprop.

Filter by
Sorted by
Tagged with
0 votes
2 answers
20 views

When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended?

When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended? I have an DNN training from an infinite stream of samples, that most likely won't ...
9 votes
1 answer
1k views

What is the formula for the momentum and Adam optimisers?

In the gradient descent algorithm, the formula to update the weight $w$, which has $g$ as the partial gradient of the loss function with respect to it, is: $$w\ -= r \times g$$ where $r$ is the ...
2 votes
1 answer
146 views

Joined vs Separate optimizer for Actor-Critic

Say that I have a simple Actor-Critic architecture, (I am not familiar with Tensorflow, but) in Pytorch we need to specify the parameters when defining an optimizer (SGD, Adam, etc) and therefore we ...
1 vote
1 answer
780 views

What is uncentered variance and how it becomes equal to mean square in Adam?

I have been reading about Adam and AdamW (Here). The author mentioned that in "uncentered variance" we don't consider subtracting mean In this statement, the author is talking about ...
1 vote
1 answer
149 views

In the update rule of RMSprop, do we divide by a matrix?

I've been trying to understand RMSprop for a long time, but there's something that keeps eluding me. Here is a screenshot from this video by Andrew Ng. From the element-wise comment, from what I ...
2 votes
1 answer
149 views

How do I use machine learning to create an optimization algorithm?

Let's say that I want to create an optimization algorithm, which is supposed to find an optimum value for a given objective function. Creating an optimization algorithm to explore through the search ...
1 vote
0 answers
114 views

Why does Adam optimizer work slower than Adagrad, Adadelta, and SGD for Neural Collaborative Filtering (NCF)?

I've been working on Neural Collaborative Filtering (NCF) recently to build a recommender system using Tensorflow Recommenders. Doing some hyperparameter tuning with different optimizers available in ...
2 votes
1 answer
179 views

What do we mean by "infrequent features"?

I am reading this blog post: https://ruder.io/optimizing-gradient-descent/index.html. In the section about AdaGrad, it says: It adapts the learning rate to the parameters, performing smaller updates (...
2 votes
3 answers
12k views

What kind of optimizer is suggested to use for binary classification of similar images?

I have spent some time searching Google and wasn't able to find out what kind of optimization algorithm is best for binary classification when images are similar to one another. I'd like to read ...