Bumped by Community user
added 13 characters in body; edited tags
Source Link
nbro
  • 31.2k
  • 8
  • 65
  • 130

Understanding multi iteration update-iteration updates of the model in the Proximal Policy Gradient PPOOptimization algorithm

I have a general question about the updating of the network/model in the PPO algorithm. 

If I understand it correctly, there are multiple iterations of weight updates done on the model with data that is created from the environment (with the model before the update). Now, I think that the updates of the model weights are not correct anymore after the first iteration/optimization step, because the model weights changed and therefore the training data is outdated (since the model would now give different actions in the environment and therefore different rewards).

Basically, in the pseudo code-code of the algorithm, I don't understand the line "Optimize surrogate L ... with K epochs...". If the update is done for multiple epochs, the data that is learned on is outdated already after the first iteration of optimization, since the modelmodel's weights changed. In other algorithms, like A2C, there is only one opimizationoptimization step done, instead of K$K$ epochs.

enter image description here

Is this some form of approximation or augmentation on the data by using the data that was created by an older model for multiple iterations or am I missing something here? If yes, where was this idea first introduced or better described? And where is aan (empirical) proof that this still leads to a correct weight updating?

Understanding multi iteration update of model in Policy Gradient PPO algorithm

I have a general question about the updating of the network/model in the PPO algorithm. If I understand it correctly, there are multiple iterations of weight updates done on the model with data that is created from the environment (with the model before the update). Now, I think that the updates of the model weights are not correct anymore after the first iteration/optimization step, because the model weights changed and therefore the training data is outdated (since the model would now give different actions in the environment and therefore different rewards).

Basically, in the pseudo code of the algorithm I don't understand the line "Optimize surrogate L ... with K epochs...". If the update is done for multiple epochs the data that is learned on is outdated already after the first iteration of optimization since the model weights changed. In other algorithms like A2C there is only one opimization step done instead of K epochs.

enter image description here

Is this some form of approximation or augmentation on the data by using the data that was created by an older model for multiple iterations or am I missing something here? If yes, where was this idea first introduced or better described? And where is a (empirical) proof that this still leads to a correct weight updating?

Understanding multi-iteration updates of the model in the Proximal Policy Optimization algorithm

I have a general question about the updating of the network/model in the PPO algorithm. 

If I understand it correctly, there are multiple iterations of weight updates done on the model with data that is created from the environment (with the model before the update). Now, I think that the updates of the model weights are not correct anymore after the first iteration/optimization step, because the model weights changed and therefore the training data is outdated (since the model would now give different actions in the environment and therefore different rewards).

Basically, in the pseudo-code of the algorithm, I don't understand the line "Optimize surrogate L ... with K epochs...". If the update is done for multiple epochs, the data that is learned is outdated already after the first iteration of optimization, since the model's weights changed. In other algorithms, like A2C, there is only one optimization step done, instead of $K$ epochs.

enter image description here

Is this some form of approximation or augmentation on the data by using the data that was created by an older model for multiple iterations or am I missing something here? If yes, where was this idea first introduced or better described? And where is an (empirical) proof that this still leads to a correct weight updating?

deleted 67 characters in body; edited tags
Source Link
nbro
  • 31.2k
  • 8
  • 65
  • 130

I have a general question about the updating of the network/model in the PPO algorithm (https://arxiv.org/abs/1707.06347PPO algorithm). If I understand it correctly, there are multiple iterations of weight updates done on the model with data that is created from the environment (with the model before the update). Now, I think that the updates of the model weights are not correct anymore after the first iteration/optimization step, because the model weights changed and therefore the training data is outdated (since the model would now give different actions in the environment and therefore different rewards).

Basically, in the pseudo code of the algorithm I don't understand the line "Optimize surrogate L ... with K epochs...". If the update is done for multiple epochs the data that is learned on is outdated already after the first iteration of optimization since the model weights changed. In other algorithms like A2C there is only one opimization step done instead of K epochs.

enter image description here

So isIs this some form of approximation or augmentation on the data by using the data that was created by an older model for multiple iterations or am I missing something here? And ifIf yes, where was this idea first introduced or better described? And where is a (empirical) proof that this still leads to a correct weight updating?

I am new to the RL field so I am sorry if this is an obvious question.

I have a general question about the updating of the network/model in the PPO algorithm (https://arxiv.org/abs/1707.06347). If I understand it correctly there are multiple iterations of weight updates done on the model with data that is created from the environment (with the model before the update). Now I think that the updates of the model weights are not correct anymore after the first iteration/optimization step because the model weights changed and therefore the training data is outdated (since the model would now give different actions in the environment and therefore different rewards).

Basically in the pseudo code of the algorithm I don't understand the line "Optimize surrogate L ... with K epochs...". If the update is done for multiple epochs the data that is learned on is outdated already after the first iteration of optimization since the model weights changed. In other algorithms like A2C there is only one opimization step done instead of K epochs.

enter image description here

So is this some form of approximation or augmentation on the data by using the data that was created by an older model for multiple iterations or am I missing something here? And if yes where was this idea first introduced or better described? And where is a (empirical) proof that this still leads to a correct weight updating?

I am new to the RL field so I am sorry if this is an obvious question.

I have a general question about the updating of the network/model in the PPO algorithm. If I understand it correctly, there are multiple iterations of weight updates done on the model with data that is created from the environment (with the model before the update). Now, I think that the updates of the model weights are not correct anymore after the first iteration/optimization step, because the model weights changed and therefore the training data is outdated (since the model would now give different actions in the environment and therefore different rewards).

Basically, in the pseudo code of the algorithm I don't understand the line "Optimize surrogate L ... with K epochs...". If the update is done for multiple epochs the data that is learned on is outdated already after the first iteration of optimization since the model weights changed. In other algorithms like A2C there is only one opimization step done instead of K epochs.

enter image description here

Is this some form of approximation or augmentation on the data by using the data that was created by an older model for multiple iterations or am I missing something here? If yes, where was this idea first introduced or better described? And where is a (empirical) proof that this still leads to a correct weight updating?

added 2 characters in body
Source Link

I have a general question about the updating of the network/model in the PPO algorithm (https://arxiv.org/abs/1707.06347). If I understand it correctly there are multiple iterations of weight updates done on the model with data that is created from the environment (with the model before the update). Now I think that the updates of the model weights are not correct anymore after the first iteration/optimization step because the model weights changed and therefore the training data is outdated (since the model would now give different actions in the environment and therefore different rewards).

Basically in the pseudo code of the algorithm I don't understand the line "Optimize surrogate L ... with K epochs...". If the update is done for multiple epochs the data that is learned on is outdated already after the first iteration of optimization since the model weights changed. In other algorithms like A2C there is only one opimization step done instead of K epochs.

enter image description here

So is this some form of approximation or augmentation on the data by using the data that was created by an older model for multiple iterations or am I missing something here? And if yes where was this idea first introduced or better described? And where is a (empirical) proof that this still leads to a correct weight updating?

I am new to thisthe RL field so I am sorry if this is an obvious question.

I have a general question about the updating of the network/model in the PPO algorithm (https://arxiv.org/abs/1707.06347). If I understand it correctly there are multiple iterations of weight updates done on the model with data that is created from the environment (with the model before the update). Now I think that the updates of the model weights are not correct anymore after the first iteration/optimization step because the model weights changed and therefore the training data is outdated (since the model would now give different actions in the environment and therefore different rewards).

Basically in the pseudo code of the algorithm I don't understand the line "Optimize surrogate L ... with K epochs...". If the update is done for multiple epochs the data that is learned on is outdated already after the first iteration of optimization since the model weights changed. In other algorithms like A2C there is only one opimization step done instead of K epochs.

enter image description here

So is this some form of approximation or augmentation on the data by using the data that was created by an older model for multiple iterations or am I missing something here? And if yes where was this idea first introduced or better described? And where is a (empirical) proof that this still leads to a correct weight updating?

I am new to this field so I am sorry if this is an obvious question.

I have a general question about the updating of the network/model in the PPO algorithm (https://arxiv.org/abs/1707.06347). If I understand it correctly there are multiple iterations of weight updates done on the model with data that is created from the environment (with the model before the update). Now I think that the updates of the model weights are not correct anymore after the first iteration/optimization step because the model weights changed and therefore the training data is outdated (since the model would now give different actions in the environment and therefore different rewards).

Basically in the pseudo code of the algorithm I don't understand the line "Optimize surrogate L ... with K epochs...". If the update is done for multiple epochs the data that is learned on is outdated already after the first iteration of optimization since the model weights changed. In other algorithms like A2C there is only one opimization step done instead of K epochs.

enter image description here

So is this some form of approximation or augmentation on the data by using the data that was created by an older model for multiple iterations or am I missing something here? And if yes where was this idea first introduced or better described? And where is a (empirical) proof that this still leads to a correct weight updating?

I am new to the RL field so I am sorry if this is an obvious question.

Source Link
Loading