3
votes
Accepted
Do the terms 'sample complexity' and 'sample efficiency' mean the same thing in RL context
The sample complexity is defined precisely in computational learning theory, which studies learning from a theoretical standpoint (like theoretical physics for physics).
Here's a definition taken from ...
- 36.3k
3
votes
Should DQN/PPO be used for state spaces that are not that large?
If I read correctly, your RL action space is a Multi-Discrete one, where each action is independent of each other and can be used simultaneously (like controller or keyboard), which is supported by ...
- 644
2
votes
In the DQN paper, why do we have both $\max_{a'}$ and $\max_{a}$ in the pseudocode?
$r_j + \gamma \max_{a'}Q(\phi_{j+1},a';\theta)$
I'm confused as to what $a'$ refers to and where it comes from.
Here $a'$ is a "dummy" argument over which you perform the maximization ...
- 1,972
2
votes
Accepted
How are these two equations for the optimal state-value function equivalent?
Your first equation is the definition of any state value function, so it must also be definition of the optimal state value function $v_*$.
The second equation is the definition of $v_*$ in terms of ...
- 36.3k
1
vote
Are the state and observation spaces essentially the same? Is it necessary to define both the state and observation space in a custom environment?
The observation space and the state space are not the same in general. There exist problems where the state space cannot be fully observed, which goes by the name ...
- 111
1
vote
Why does A2C use the actual returns from an episode in calculating the advantage?
Let's look at how they derived the equation for that. Recall that $A(s,a) = Q(s,a) - V(s)$, so in the first equation, it is $Q(s,a) = \sum_t \gamma^tr_t$. We should noted that, the original Policy ...
- 644
1
vote
How to setup a reinforcement learning model that changes the values of $x$ to maximize $y$ in $y = f(x)$?
Sure your idea makes sense to me. Yes you can give y as the reward.
No need to control for the fact that the input has only discrete values 0 and 1. Technically, most RL codes would normalize the ...
- 1,111
1
vote
Accepted
Should I use multi-armed-bandits or RL for a financial time-series problem?
Your agent's actions will (probably) not have much impact on the observed financial time series. However, they will make a large difference to other things - namely what stock your agent is holding ...
- 25.4k
1
vote
How can imitation learning data be collected?
Imitation learning data usually means data gathered from an expert, that is data from an agent proficient in the task.
The agent may be:
A human operator: have the operator complete the task and ...
- 583
1
vote
Reinforcement Learning with PPO - entropy loss dropping, but so is performance. Why?
PPO is an algorithm in a class of actor-critic methods. In this class of methods, the training is broken in two counteracting parts: the "actor" parts learns the policy and the "critic&...
- 1,972
1
vote
What is the difference between an on-policy distribution and state visitation frequency?
1.First of all. The on-policy distribution $\mu(s)$ is a probability distribution. So, obviously, it is different from state visitation frequency $\rho_\pi(s)$, since $\rho_\pi(s)$ is not normalized ...
- 1,972
1
vote
What introductory books to reinforcement learning do you know, and how do they approach this topic?
The (draft) book Reinforcement Learning: Theory and Algorithms, by Sham M. Kakade (who published a natural policy gradient algorithm and other important research) and others, introduces RL in a ...
- 36.3k
Only top scored, non community-wiki answers of a minimum length are eligible