Questions tagged [sample-efficiency]
For questions about the sample efficiency (or inefficiency) of learning algorithms, which is the amount of experience (or data) that the learning algorithm needs in order to reach a certain level of performance. In the case of reinforcement learning, this experience is represented by (transition) tuples of the form $(s, a, r, s')$.
16
questions
2
votes
1
answer
245
views
Do the terms 'sample complexity' and 'sample efficiency' mean the same thing in RL context
For example, the the paper Soft Actor-Critic:Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, both terms are mentioned but without explaining. I have seen them in other ...
- 175
2
votes
0
answers
115
views
Why do off-policy algorithms suffer from worse computational or time efficiency compared to on-policy algorithms?
When I run Soft-Actor-Critic (off-policy) in my Environment, the calculation of gradient updates takes almost twice the time compared to using PPO (on-policy).
I also saw that ACER has a higher time ...
- 115
1
vote
1
answer
99
views
If Least-Squares TD is computationally more expensive, then why is it more data efficient than semi-gradient TD(0)?
In Sutton-Barto (Section: 9.8 Least-Squares TD, page 228):
Authors say that Least-Squares TD is the most "data efficient" form of linear TD(0). Later, in this section, they say the ...
- 179
1
vote
1
answer
298
views
Does importance sampling really improve sampling efficiency of TRPO or PPO?
Vanilla policy gradient has a loss function:
$$\mathcal{L}_{\pi_{\theta}(\theta)} = E_{\tau \sim \pi_{\theta}}[\sum\limits_{t = 0}^{\infty}\gamma^{t}r_{t}]$$
while in TRPO it is:
$$\mathcal{L}_{\pi_{\...
- 41
2
votes
1
answer
476
views
Reinforcement Learning method suitable for a large discrete action space with high sample efficiency
Consider the following problem.
We have a process, that generates $N$ stones (e.g. 2000) in one batch $b$. Every pebble has state $s_{i}^b$ and reward $s_i^b$. After choosing one pebble $i$ from the $...
- 323
1
vote
1
answer
94
views
How to use a heuristic policy to increase sample efficiency of a deep reinforcement learning agent?
I have a heuristic solution to a problem which works quite well when certain environmental parameters are known and unchanging. However, in a real world setting these parameters will not be known and ...
- 153
0
votes
0
answers
47
views
Does randomly adding hand-engineered features increase the CNN's sample efficiency/performance?
It is a known fact that preprocessing images using CV techniques will improve CNN performance (see this answer).
But what happens when you feed in the entire image and the filtered image randomly to ...
- 588
3
votes
1
answer
213
views
How do I represent sample efficiency of RL rewards in mathematical notation?
I define sample efficiency as the area under the curve/graph, where $x$-axis is the number of episodes while y-axis is the cumulative reward for that episode. I would like to formally define it with a ...
- 31
1
vote
0
answers
46
views
How can I estimate the minimum number of training samples needed to get interesting results with WGAN?
Let's say we have a WGAN where the generator and critic have 8 layers and 5 million parameters each. I know that the greater the number of training samples the better, but is there a way to know the ...
7
votes
1
answer
1k
views
How to measure sample efficiency of a reinforcement learning algorithm?
I want to know if there is any metric to use for measuring sample-efficiency of a reinforcement learning algorithm? From reading research papers, I see claims that proposed models are more sample ...
- 320
2
votes
0
answers
117
views
Can you find another reason for sample inefficiency of model-free on-policy Deep Reinforcement Learning?
The following mindmap gives an overview of multiple reasons for sample inefficiency. The list is definitely not complete. Can you see another reason not mentioned so far?
Some related links:
...
- 451
1
vote
0
answers
99
views
Should we start with a small batch-size and increase during training to improve sample efficiency?
Just made an interesting observation playing around with the stable-baseline's implementation of PPO and the BipedalWalker environment from OpenAI's Gym. But I believe this should be a general ...
- 451
7
votes
2
answers
1k
views
Why are reinforcement learning methods sample inefficient?
Reinforcement learning methods are considered to be extremely sample inefficient.
For example, in a recent DeepMind paper by Hessel et al., they showed that in order to reach human-level performance ...
- 263
4
votes
1
answer
2k
views
Why are model-based methods more sample efficient than model-free methods?
Why do model-based methods use fewer samples than model-free methods? Here, I'm specifically referring to model-based methods in which we have to learn a policy and model. I can only think of two ...
- 441
27
votes
2
answers
17k
views
What is sample efficiency, and how can importance sampling be used to achieve it?
For instance, the title of this paper reads: "Sample Efficient Actor-Critic with Experience Replay".
What is sample efficiency, and how can importance sampling be used to achieve it?
- 423
6
votes
2
answers
1k
views
What is the current state-of-the-art in Reinforcement Learning regarding data efficiency?
In other words, which existing reinforcement method learns with fewest episodes? R-Max comes to mind, but it's very old and I'd like to know if there is something better now.
- 2,089