# Questions tagged [optimizers]

For questions about optimization methods/algorithms (also know as optimizers) in the context of machine learning and other AI subfields. Examples of optimizers are plain (stochastic) gradient descent, Adam, SGD with momentum, Adagrad, and RMSprop.

9 questions
Filter by
Sorted by
Tagged with
17 views

### Is the Adam optimizer moment values different for each layer?

If I have an 3 tensorflow layers in a network and the 2 weights between these layers are of different dimensions, how does the Adam optimizer algorithm work? For the pseudocode in https://optimization....
90 views

### How do I use machine learning to create an optimization algorithm?

Let's say that I want to create an optimization algorithm, which is supposed to find an optimum value for a given objective function. Creating an optimization algorithm to explore through the search ...
1 vote
359 views

### What is uncentered variance and how it becomes equal to mean square in Adam?

I have been reading about Adam and AdamW (Here). The author mentioned that in "uncentered variance" we don't consider subtracting mean In this statement, the author is talking about ...
• 151
1 vote
60 views

### Joined vs Separate optimizer for Actor-Critic

Say that I have a simple Actor-Critic architecture, (I am not familiar with Tensorflow, but) in Pytorch we need to specify the parameters when defining an optimizer (SGD, Adam, etc) and therefore we ...
• 143
1 vote
31 views

I've been working on Neural Collaborative Filtering (NCF) recently to build a recommender system using Tensorflow Recommenders. Doing some hyperparameter tuning with different optimizers available in ...
1 vote
85 views

### In the update rule of RMSprop, do we divide by a matrix?

I've been trying to understand RMSprop for a long time, but there's something that keeps eluding me. Here is a screenshot from this video by Andrew Ng. From the element-wise comment, from what I ...
129 views

### What do we mean by "infrequent features"?

1 vote
In the gradient descent algorithm, the formula to update the weight $w$, which has $g$ as the partial gradient of the loss function with respect to it, is: $$w\ -= r \times g$$ where $r$ is the ...