New answers tagged reinforcement-learning
0
votes
Unclear point in TRPO
In practice policy optimization algorithms like TRPO collect trajectories of experiences by interacting with the environment under the current policy, thus the sloppy notation here with expectation ...
- 3,069
0
votes
Derivation of the relationship between state-value function and action-value function in SAC
This relation is from the definition of action value function such as that defined in Sutton's RL book page 58.
we define the value of taking action $a$ in state $s$ under a policy $\pi$, denoted
$q_{...
- 3,069
0
votes
Why does Multi Objective RL exist?
Yes, we can do that (look at the examples from MORL Gymnasium). They have a linear function to weight more rewards into one scalar and they use only this value as any single policy single reward ...
- 204
0
votes
How Do I Optimise a Black-Box Objective Function with DQN Using Reinforcement Learning?
Since you mentioned that you are a beginner in RL, I highly suggest that you engineer the environment to be compatible with off-the-shelf RL libraries to compare with your implemented DQN algorithm. ...
- 1,568
4
votes
Can reinforcement learning rewards be a combination of current and new state?
The equation you presented is the expected reward function, which will always produce the same scalar value for a specific $(s, a, s')$ tuple. An essential concern when designing the reward signal for ...
- 1,568
4
votes
Can reinforcement learning rewards be a combination of current and new state?
Yes, it is valid within the MDP framework to base immediate reward on current state, action and next state, plus a random factor. Or any subset and combination.
The important detail is that the ...
- 32.9k
0
votes
How feasible is it to automate Theorem Proving via Reinforcement Learning?
It is feasible, even though we are at the level of high-school mathematics at the moment. But there is room for scaling. Examples are:
HyperTree Proof Search for Neural Theorem Proving (2022)
DeepSeek-...
- 1,174
1
vote
Accepted
Unclear derivation for the PPO method
The first line of equation (11.4) is copied from equation (11.2) where the definition of your book's advantage function $\hat A_{\pi}(s_t,a_t)$ plays a key role and it seems you have no problem for ...
- 3,069
3
votes
Accepted
Unclear point in definition of advantage function in PPO
It's defined this way so that you can have the above equational relation between the $\eta$-functions of the old and the current policies, as proved in slide 19. And on page 20, the improvement theory ...
- 3,069
1
vote
Accepted
Is this actor-critic algorithm correct?
In S&B RL book page 332 pseudocode for one-step actor-critic method with 2 nested loops, both policy and critic parameters could be initialized arbitrarily, and for example they're both ...
- 3,069
-1
votes
Interpretation of changing action probability based on policy gradient expression
Bear in mind $\nabla_{\theta} J$ is used to update parameters $\theta$ in each step and $r(\tau)$ is the total reward of a sampling trajectory $\tau$ where we can rewrite the policy gradient equation ...
- 3,069
2
votes
Accepted
Why in loss function of DQN does the expectation depend on current state $s$?
In DQN the loss function is designed to minimize the difference between the prediction network's $Q(s,a;\theta)$ and the target network's $Q^*(s,a)$ computed using the Bellman optimality equation as a ...
- 3,069
2
votes
Accepted
Questions about notation in RL
Yes, $S_t \sim d^\pi$ is a nice way of saying that the states are distributed according to the state distribution induced by following $\pi$. Whilst $\pi$ does not directly choose the next state, ...
- 5,000
1
vote
Accepted
Material suggestion for policy gradient methods in reinforcement learning?
I like these explanations:
https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html
https://lilianweng.github.io/posts/2018-04-08-policy-gradient/
I think you can find what you seek in both ...
- 124
0
votes
How does Importance Sampling compare against Policy Regularization in Offline RL?
One way to handle this, is to keep the target policy close to the
behavior policy by regularizing, for example by using a KL divergence
penalty. So when the target policy deviates too much from the ...
- 1,568
2
votes
Accepted
How to define a good reward function to keep output $y$ in the range $[y_{min}, y_{max}]$?
There are multiple ways to structure a reward system to manage some form of homeostasis, i.e. where despite changes in external environment or forces acting against the agent, it takes action to ...
- 32.9k
1
vote
Do State Variables in RL Models Need Direct Update Equations?
The state of an RL agent, in the naive case, just has to be a comprehensive description of the system, or at least the most comprehensive available (in some cases, it's impossible to have a complete ...
- 2,473
Top 50 recent answers are included