Questions tagged [soft-actor-critic]
For questions about Soft Actor-Critic (SAC), which was proposed in "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor" by Tuomas Haarnoja et al.
31
questions
-1
votes
0
answers
18
views
Can someone include a link to an article that can help me learn soft actor-critic?
I've been looking through articles on Medium and just papers in general. I can't seem to find anything that really makes sense to me. I always learn better from videos, but there are barely any videos ...
- 109
0
votes
0
answers
12
views
Help defining environment with complex action space
I'm working on a personal MARL project with a high-dimensional and continuous action space. The environment is designed to give positive rewards to actions between some moving limits of the action ...
0
votes
0
answers
20
views
Training Issue in Solving Multi-Dimensional Multiple Knapsack Problem with Transformer Model and PPO and SAC algorithm
I'm reaching out to the brilliant minds of the AI community to seek help with a challenging issue in my project on solving the multi-dimensional multiple knapsack problem using a transformer model. As ...
1
vote
1
answer
48
views
Why do we limit the standard deviation in Actor architectures in Reinforcement Learning?
I'm in the process of implementing Actor-Critic structures for Reinforcement Learning (RL) and I've noticed that it's a common practice to limit the standard deviation (std). I've seen this in ...
- 41
0
votes
1
answer
143
views
Where does the term $\log \mu(u \mid s)$ come from?
This question comes from trying to build a SAC model. The action space is derived from a log normal distribution.
If in the appendix c of the original paper the equation for the log policy is:
$\log \...
- 15
0
votes
1
answer
55
views
Why clamp std for reparameterization trick between -20 and 2?
In the Soft Actor Critic Paper (found here https://arxiv.org/pdf/1801.01290.pdf), they use a neural network to approximate a diagonal gaussian distribution. In the sample function you can see that it ...
- 15
2
votes
1
answer
223
views
Why is soft actor critic an off policy scheme?
I am struggling to understand what makes a scheme on-policy or off-policy. From what I have read, we can say that deep Q-learning is off-policy because we use a different policy like $\epsilon$-greedy ...
- 35
1
vote
1
answer
195
views
Why is Soft Q Learning not an Actor Critic method?
I've been reading these two papers from Haarnoja et. al.:
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement
Learning with a Stochastic Actor
Reinforcement Learning with Deep Energy-...
1
vote
1
answer
50
views
Can action be dominated by state features in actor-critic algorithms?
I have a case where my state consists of relatively large number of features, e.g. 50, whereas my action size is 1. I wonder whether my state features dominate the action in my critic network. I ...
- 331
0
votes
0
answers
121
views
Why do Soft Actor-Critic with automatic temperature tuning use only a single dual variable?
In section 5 of the paper “Soft Actor Critic Algorithms and Applications”, the authors propose to optimize the policy subject to the constraints that the entropy of action distribution should be ...
- 203
0
votes
0
answers
72
views
Why don't SAC implementations use state value function?
When I read SAC paper, they use state value function $V_\psi(s_t)$ and as do their implementation. But in other SAC implementations like stable-baseline3, pytorch-soft-actor-critic, it seems that they ...
- 1
1
vote
0
answers
298
views
Why Soft Actor-Critic (SAC) uses a double Q trick?
Twin Delayed DDPG (TD3) uses a double Q trick since the policy is deterministic like in DDPG, which is to mitigate the maximum overestimation bias in DDPG. However, in SAC, the policy is stochastic, ...
- 41
1
vote
1
answer
167
views
Can RL still learn if part of my actions are only used once, at the beginning of the episode?
I am working in an environment with 3-dimensional action space. The first two actions are only used at the first timestep and never again. The third action is used at every timestep.
Say, the action ...
- 13
0
votes
1
answer
300
views
How to build the actor policy of Soft-Actor-Critic after sampling from a Multivariate normal distribution?
I'm trying to solve LunarLanderContinuous-v2 (https://www.gymlibrary.ml/environments/box2d/lunar_lander/) using Soft Actor-Critic algorithm (following the pseudocode above)
To update the actor policy ...
- 13
1
vote
0
answers
47
views
Can entropy bonus be used with state-independent log std for stochastic policies?
In this blog article by openai, they say the std of the exploration distribution must be state-dependent, i.e. an output of the policy network, so it works with the entropy bonus, which is an integral ...
- 131
0
votes
0
answers
20
views
How to establish baselines, with different training loops
My objective is to test out a new algorithm that I designed. However, I am confused whether my methodology to train the networks is correct.
I am just concerned about the training loops:
In the first ...
- 1
3
votes
1
answer
4k
views
Does SAC perform better than PPO in sample-expensive tasks with discrete action spaces?
I am currently using Proximal Policy Optimization (PPO) to solve my RL task. However, after reading about Soft Actor-Critic (SAC) now I am unsure whether I should stick to PPO or switch to SAC. ...
- 53
1
vote
0
answers
26
views
Why exclude the first entropy bonus term in the soft Q-function in SAC?
Based on OpenAI Spinning Up description of Soft Actor Critic (SAC) the soft Q-function is defined as
and as they say
Q value is changed to include the entropy bonuses from every timestep except the ...
- 111
2
votes
1
answer
227
views
Can off-policy algorithms benefit from the parallelization?
On-policy algorithms, such as A2C, A3C and PPO, leverage massive parallelization to achieve state of the art results. However, I’ve never come across parallelization efforts when it comes to the off-...
- 331
1
vote
0
answers
53
views
How to compare RL algorithms with different NN sizes?
I wanted to run some tests with some RL algorithms in a continuous control task, namely PPO-clip and SAC.
When comparing their NN structures described in their papers, SAC used 2 layers with 256 ...
- 115
0
votes
0
answers
257
views
Is it possible to use Softmax as an activation function for actor (policy) network in TD3 or SAC Reinforcement learning algorithms?
As I understand from literature, normally, the last activation in an actor (policy) network in TD3 and SAC algorithms is a Tanh function, which is scaled by a certain limit.
My action vector is ...
- 101
3
votes
0
answers
1k
views
How to interpret the training loss curves in Soft-Actor-Critic (SAC)?
I am using stable-baseline3 implementation of the Soft-Actor-Critic (SAC) algorithm. The plotted training curves look promising. However, I am not fully sure how to interpret the actor and critic ...
- 45
1
vote
1
answer
379
views
How to enforce action bounds between 0 & 1 in soft actor-critic algorithm?
In the paper "Soft Actor-Critic Algorithms and Applications", appendix C shows enforcing action bounds using the tanh squashing function which is in (-1, 1). I have action bounds in (0, 1), ...
- 331
3
votes
0
answers
2k
views
Optimal episode length in reinforcement learning
I have a custom environment for stock trading where an episode can be as long as 2000-3000 steps. I've run several experiments with td3 and sac algorithms, average reward per episode flattens after ...
- 331
0
votes
0
answers
48
views
How is the discounted maximum entropy objective obtained for soft-q-learning and SAC
In the soft q-learning paper, they provide an expression for the maximum entropy objective that takes discounting into account.
My main question is: can someone explain how they incorporated ...
- 384
1
vote
0
answers
279
views
How to make SAC (Soft-Actor-Critic) learn a policy?
I cannot make SAC learn a task in a certain environment. The point is that it actually sometimes finds a very good policy, but it never learns the policy in the end. I am using the SAC implementation ...
- 11
3
votes
0
answers
517
views
In Soft Actor Critic, why is the action sampled from current policy instead of replay buffer on value function update?
While reading the original paper of Soft Actor Critic, I came across on page number 5, under equation (5) and (6)
$$
J_{V}(\psi)=\mathbb{E}_{\mathbf{s}_{t} \sim \mathcal{D}}\left[\frac{1}{2}\left(V_{\...
- 31
0
votes
1
answer
1k
views
Why is my Soft Actor-Critic's policy and value function losses not converging?
I'm trying to implement a soft actor-critic algorithm for financial data (stock prices), but I have trouble with losses: no matter what combination of hyper-parameters I enter, they are not converging,...
- 3
3
votes
1
answer
499
views
Where does entropy enter in Soft Actor-Critic?
I am currently trying to understand SAC (Soft Actor-Critic), and I am thinking of it as a basic actor-critic with the entropy included. However, I expected the entropy to appear in the Q-function. ...
- 117
1
vote
0
answers
418
views
How does the automated temperature adjustment step work in Soft Actor-Critic?
In section 5 of the paper Soft Actor-Critic Algorithms and Applications, it is proposed an optimization problem to obtain an optimal temperature parameter $\alpha^*_t$. First, one uses the original ...
- 395
3
votes
2
answers
513
views
What is the gradient of the objective function in the Soft Actor-Critic paper?
In the paper Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, they define the loss function for the policy network as
$$
J_\pi(\phi)=\mathbb E_{s_t\...
- 441