Questions tagged [sample-efficiency]

For questions about the sample efficiency (or inefficiency) of learning algorithms, which is the amount of experience (or data) that the learning algorithm needs in order to reach a certain level of performance. In the case of reinforcement learning, this experience is represented by (transition) tuples of the form $(s, a, r, s')$.

Filter by
Sorted by
Tagged with
3 votes
1 answer
434 views

Do the terms 'sample complexity' and 'sample efficiency' mean the same thing in RL context

For example, the the paper Soft Actor-Critic:Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, both terms are mentioned but without explaining. I have seen them in other ...
Sam's user avatar
  • 185
2 votes
0 answers
135 views

Why do off-policy algorithms suffer from worse computational or time efficiency compared to on-policy algorithms?

When I run Soft-Actor-Critic (off-policy) in my Environment, the calculation of gradient updates takes almost twice the time compared to using PPO (on-policy). I also saw that ACER has a higher time ...
kitaird's user avatar
  • 115
1 vote
1 answer
110 views

If Least-Squares TD is computationally more expensive, then why is it more data efficient than semi-gradient TD(0)?

In Sutton-Barto (Section: 9.8 Least-Squares TD, page 228): Authors say that Least-Squares TD is the most "data efficient" form of linear TD(0). Later, in this section, they say the ...
user3489173's user avatar
1 vote
1 answer
462 views

Does importance sampling really improve sampling efficiency of TRPO or PPO?

Vanilla policy gradient has a loss function: $$\mathcal{L}_{\pi_{\theta}(\theta)} = E_{\tau \sim \pi_{\theta}}[\sum\limits_{t = 0}^{\infty}\gamma^{t}r_{t}]$$ while in TRPO it is: $$\mathcal{L}_{\pi_{\...
Magi Feeney's user avatar
2 votes
1 answer
512 views

Reinforcement Learning method suitable for a large discrete action space with high sample efficiency

Consider the following problem. We have a process, that generates $N$ stones (e.g. 2000) in one batch $b$. Every pebble has state $s_{i}^b$ and reward $s_i^b$. After choosing one pebble $i$ from the $...
Daniel Wiczew's user avatar
1 vote
1 answer
101 views

How to use a heuristic policy to increase sample efficiency of a deep reinforcement learning agent?

I have a heuristic solution to a problem which works quite well when certain environmental parameters are known and unchanging. However, in a real world setting these parameters will not be known and ...
asheets's user avatar
  • 153
0 votes
0 answers
48 views

Does randomly adding hand-engineered features increase the CNN's sample efficiency/performance?

It is a known fact that preprocessing images using CV techniques will improve CNN performance (see this answer). But what happens when you feed in the entire image and the filtered image randomly to ...
desert_ranger's user avatar
3 votes
1 answer
228 views

How do I represent sample efficiency of RL rewards in mathematical notation?

I define sample efficiency as the area under the curve/graph, where $x$-axis is the number of episodes while y-axis is the cumulative reward for that episode. I would like to formally define it with a ...
bluewander's user avatar
1 vote
0 answers
48 views

How can I estimate the minimum number of training samples needed to get interesting results with WGAN?

Let's say we have a WGAN where the generator and critic have 8 layers and 5 million parameters each. I know that the greater the number of training samples the better, but is there a way to know the ...
FalseSemiColon's user avatar
7 votes
1 answer
1k views

How to measure sample efficiency of a reinforcement learning algorithm?

I want to know if there is any metric to use for measuring sample-efficiency of a reinforcement learning algorithm? From reading research papers, I see claims that proposed models are more sample ...
rert588's user avatar
  • 320
2 votes
0 answers
127 views

Can you find another reason for sample inefficiency of model-free on-policy Deep Reinforcement Learning?

The following mindmap gives an overview of multiple reasons for sample inefficiency. The list is definitely not complete. Can you see another reason not mentioned so far? Some related links: ...
Ray Walker's user avatar
1 vote
0 answers
162 views

Should we start with a small batch-size and increase during training to improve sample efficiency?

Just made an interesting observation playing around with the stable-baseline's implementation of PPO and the BipedalWalker environment from OpenAI's Gym. But I believe this should be a general ...
Ray Walker's user avatar
7 votes
2 answers
2k views

Why are reinforcement learning methods sample inefficient?

Reinforcement learning methods are considered to be extremely sample inefficient. For example, in a recent DeepMind paper by Hessel et al., they showed that in order to reach human-level performance ...
rrz0's user avatar
  • 263
5 votes
1 answer
2k views

Why are model-based methods more sample efficient than model-free methods?

Why do model-based methods use fewer samples than model-free methods? Here, I'm specifically referring to model-based methods in which we have to learn a policy and model. I can only think of two ...
Maybe's user avatar
  • 461
31 votes
2 answers
19k views

What is sample efficiency, and how can importance sampling be used to achieve it?

For instance, the title of this paper reads: "Sample Efficient Actor-Critic with Experience Replay". What is sample efficiency, and how can importance sampling be used to achieve it?
Gokul NC's user avatar
  • 463
6 votes
2 answers
1k views

What is the current state-of-the-art in Reinforcement Learning regarding data efficiency?

In other words, which existing reinforcement method learns with fewest episodes? R-Max comes to mind, but it's very old and I'd like to know if there is something better now.
rcpinto's user avatar
  • 2,119