# Tag Info

### Unclear point in TRPO

In practice policy optimization algorithms like TRPO collect trajectories of experiences by interacting with the environment under the current policy, thus the sloppy notation here with expectation ...
• 3,069

This relation is from the definition of action value function such as that defined in Sutton's RL book page 58. we define the value of taking action $a$ in state $s$ under a policy $\pi$, denoted $q_{... • 3,069 0 votes ### Why does Multi Objective RL exist? Yes, we can do that (look at the examples from MORL Gymnasium). They have a linear function to weight more rewards into one scalar and they use only this value as any single policy single reward ... • 204 0 votes ### How Do I Optimise a Black-Box Objective Function with DQN Using Reinforcement Learning? Since you mentioned that you are a beginner in RL, I highly suggest that you engineer the environment to be compatible with off-the-shelf RL libraries to compare with your implemented DQN algorithm. ... • 1,568 4 votes ### Can reinforcement learning rewards be a combination of current and new state? The equation you presented is the expected reward function, which will always produce the same scalar value for a specific$(s, a, s')$tuple. An essential concern when designing the reward signal for ... • 1,568 4 votes ### Can reinforcement learning rewards be a combination of current and new state? Yes, it is valid within the MDP framework to base immediate reward on current state, action and next state, plus a random factor. Or any subset and combination. The important detail is that the ... • 32.9k 0 votes ### How feasible is it to automate Theorem Proving via Reinforcement Learning? It is feasible, even though we are at the level of high-school mathematics at the moment. But there is room for scaling. Examples are: HyperTree Proof Search for Neural Theorem Proving (2022) DeepSeek-... • 1,174 1 vote Accepted ### Unclear derivation for the PPO method The first line of equation (11.4) is copied from equation (11.2) where the definition of your book's advantage function$\hat A_{\pi}(s_t,a_t)$plays a key role and it seems you have no problem for ... • 3,069 3 votes Accepted ### Unclear point in definition of advantage function in PPO It's defined this way so that you can have the above equational relation between the$\eta$-functions of the old and the current policies, as proved in slide 19. And on page 20, the improvement theory ... • 3,069 1 vote Accepted ### Is this actor-critic algorithm correct? In S&B RL book page 332 pseudocode for one-step actor-critic method with 2 nested loops, both policy and critic parameters could be initialized arbitrarily, and for example they're both ... • 3,069 -1 votes ### Interpretation of changing action probability based on policy gradient expression Bear in mind$\nabla_{\theta} J$is used to update parameters$\theta$in each step and$r(\tau)$is the total reward of a sampling trajectory$\tau$where we can rewrite the policy gradient equation ... • 3,069 2 votes Accepted ### Why in loss function of DQN does the expectation depend on current state$s$? In DQN the loss function is designed to minimize the difference between the prediction network's$Q(s,a;\theta)$and the target network's$Q^*(s,a)$computed using the Bellman optimality equation as a ... • 3,069 2 votes Accepted ### Questions about notation in RL Yes,$S_t \sim d^\pi$is a nice way of saying that the states are distributed according to the state distribution induced by following$\pi$. Whilst$\pi$does not directly choose the next state, ... • 5,000 1 vote Accepted ### Material suggestion for policy gradient methods in reinforcement learning? I like these explanations: https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html https://lilianweng.github.io/posts/2018-04-08-policy-gradient/ I think you can find what you seek in both ... 0 votes ### How does Importance Sampling compare against Policy Regularization in Offline RL? One way to handle this, is to keep the target policy close to the behavior policy by regularizing, for example by using a KL divergence penalty. So when the target policy deviates too much from the ... • 1,568 2 votes Accepted ### How to define a good reward function to keep output$y$in the range$[y_{min}, y_{max}]\$?

There are multiple ways to structure a reward system to manage some form of homeostasis, i.e. where despite changes in external environment or forces acting against the agent, it takes action to ...
• 32.9k
1 vote

### Do State Variables in RL Models Need Direct Update Equations?

The state of an RL agent, in the naive case, just has to be a comprehensive description of the system, or at least the most comprehensive available (in some cases, it's impossible to have a complete ...
• 2,473

Top 50 recent answers are included