Questions tagged [optimizers]

For questions about optimization methods/algorithms (also know as optimizers) in the context of machine learning and other AI subfields. Examples of optimizers are plain (stochastic) gradient descent, Adam, SGD with momentum, Adagrad, and RMSprop.

4 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
1
vote
0answers
16 views

Why does Adam optimizer work slower than Adagrad, Adadelta, and SGD for Neural Collaborative Filtering (NCF)?

I've been working on Neural Collaborative Filtering (NCF) recently to build a recommender system using Tensorflow Recommenders. Doing some hyperparameter tuning with different optimizers available in ...
0
votes
0answers
16 views

Joined vs Separate optimizer for Actor-Critic

Say that I have a simple Actor-Critic architecture, (I am not familiar with Tensorflow, but) in Pytorch we need to specify the parameters when defining an optimizer (SGD, Adam, etc) and therefore we ...
0
votes
0answers
22 views

How is the marginal likelihood of wide valleys higher than that of narrow valleys when optimizing a cost function?

I am reading the paper Entropy-sgd: Biasing gradient descent into wide valleys by Chaudhari et al. From what I understand, wide valleys tend to generalize better than sharp ones because they are ...
0
votes
1answer
55 views

RMSprop equation - dividing by a matrix?

I've been trying to understand RMSprop for a long time, but there's something that keeps eluding me. $dW$ and $db$ are matrices (that's what I understand from the element-wise comment), so that must ...