Skip to main content

Questions tagged [soft-actor-critic]

For questions about Soft Actor-Critic (SAC), which was proposed in "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor" by Tuomas Haarnoja et al.

Filter by
Sorted by
Tagged with
1 vote
1 answer
49 views

Why soft actor critic uses exponential of Q when updating policy? and what is a partition function?

Soft Actor-Critic paper proposes $\pi_{new}=\arg\min_{\pi'\in \prod}D_{KL}\left(\pi'(\cdot|s_t)\big|\big| \frac{\exp(Q^{\pi_{old}}(s_t,\cdot))}{Z^{\pi_{old}}(s_t)}\right)$ Paper says, we update the ...
user3315463's user avatar
0 votes
0 answers
10 views

How to clip actions based on state in RL environment for trading (and other tips to approach optimal trade execution)?

I’m currently trying to use RL to analyze price impact in financial markets for optimal trade execution and have coded up a custom gymnasium environment to do so. Now, I'm deciding on which RL ...
walrus's user avatar
  • 1
0 votes
0 answers
28 views

Help defining environment with complex action space

I'm working on a personal MARL project with a high-dimensional and continuous action space. The environment is designed to give positive rewards to actions between some moving limits of the action ...
Sebastian Tinoco's user avatar
0 votes
0 answers
37 views

Training Issue in Solving Multi-Dimensional Multiple Knapsack Problem with Transformer Model and PPO and SAC algorithm

I'm reaching out to the brilliant minds of the AI community to seek help with a challenging issue in my project on solving the multi-dimensional multiple knapsack problem using a transformer model. As ...
Mohammad Hosseini's user avatar
2 votes
1 answer
134 views

Why do we limit the standard deviation in Actor architectures in Reinforcement Learning?

I'm in the process of implementing Actor-Critic structures for Reinforcement Learning (RL) and I've noticed that it's a common practice to limit the standard deviation (std). I've seen this in ...
XiaoBanni's user avatar
0 votes
1 answer
235 views

Where does the term $\log \mu(u \mid s)$ come from?

This question comes from trying to build a SAC model. The action space is derived from a log normal distribution. If in the appendix c of the original paper the equation for the log policy is: $\log \...
chadmc's user avatar
  • 15
0 votes
1 answer
92 views

Why clamp std for reparameterization trick between -20 and 2?

In the Soft Actor Critic Paper (found here https://arxiv.org/pdf/1801.01290.pdf), they use a neural network to approximate a diagonal gaussian distribution. In the sample function you can see that it ...
chadmc's user avatar
  • 15
2 votes
1 answer
352 views

Why is soft actor critic an off policy scheme?

I am struggling to understand what makes a scheme on-policy or off-policy. From what I have read, we can say that deep Q-learning is off-policy because we use a different policy like $\epsilon$-greedy ...
ZZ1's user avatar
  • 35
1 vote
1 answer
413 views

Why is Soft Q Learning not an Actor Critic method?

I've been reading these two papers from Haarnoja et. al.: Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor Reinforcement Learning with Deep Energy-...
frances_farmer's user avatar
1 vote
1 answer
62 views

Can action be dominated by state features in actor-critic algorithms?

I have a case where my state consists of relatively large number of features, e.g. 50, whereas my action size is 1. I wonder whether my state features dominate the action in my critic network. I ...
Mika's user avatar
  • 341
0 votes
0 answers
187 views

Why do Soft Actor-Critic with automatic temperature tuning use only a single dual variable?

In section 5 of the paper “Soft Actor Critic Algorithms and Applications”, the authors propose to optimize the policy subject to the constraints that the entropy of action distribution should be ...
Cloudy's user avatar
  • 223
0 votes
0 answers
112 views

Why don't SAC implementations use state value function?

When I read SAC paper, they use state value function $V_\psi(s_t)$ and as do their implementation. But in other SAC implementations like stable-baseline3, pytorch-soft-actor-critic, it seems that they ...
DevSlem's user avatar
1 vote
0 answers
459 views

Why Soft Actor-Critic (SAC) uses a double Q trick?

Twin Delayed DDPG (TD3) uses a double Q trick since the policy is deterministic like in DDPG, which is to mitigate the maximum overestimation bias in DDPG. However, in SAC, the policy is stochastic, ...
Magi Feeney's user avatar
1 vote
1 answer
228 views

Can RL still learn if part of my actions are only used once, at the beginning of the episode?

I am working in an environment with 3-dimensional action space. The first two actions are only used at the first timestep and never again. The third action is used at every timestep. Say, the action ...
user13018205's user avatar
0 votes
1 answer
368 views

How to build the actor policy of Soft-Actor-Critic after sampling from a Multivariate normal distribution?

I'm trying to solve LunarLanderContinuous-v2 (https://www.gymlibrary.ml/environments/box2d/lunar_lander/) using Soft Actor-Critic algorithm (following the pseudocode above) To update the actor policy ...
Luc-dotcom's user avatar
1 vote
0 answers
55 views

Can entropy bonus be used with state-independent log std for stochastic policies?

In this blog article by openai, they say the std of the exploration distribution must be state-dependent, i.e. an output of the policy network, so it works with the entropy bonus, which is an integral ...
flxh's user avatar
  • 131
0 votes
0 answers
20 views

How to establish baselines, with different training loops

My objective is to test out a new algorithm that I designed. However, I am confused whether my methodology to train the networks is correct. I am just concerned about the training loops: In the first ...
Yash_Bit's user avatar
3 votes
1 answer
7k views

Does SAC perform better than PPO in sample-expensive tasks with discrete action spaces?

I am currently using Proximal Policy Optimization (PPO) to solve my RL task. However, after reading about Soft Actor-Critic (SAC) now I am unsure whether I should stick to PPO or switch to SAC. ...
Aeryan's user avatar
  • 53
1 vote
0 answers
32 views

Why exclude the first entropy bonus term in the soft Q-function in SAC?

Based on OpenAI Spinning Up description of Soft Actor Critic (SAC) the soft Q-function is defined as and as they say Q value is changed to include the entropy bonuses from every timestep except the ...
Daniel's user avatar
  • 111
2 votes
1 answer
332 views

Can off-policy algorithms benefit from the parallelization?

On-policy algorithms, such as A2C, A3C and PPO, leverage massive parallelization to achieve state of the art results. However, I’ve never come across parallelization efforts when it comes to the off-...
Mika's user avatar
  • 341
1 vote
0 answers
58 views

How to compare RL algorithms with different NN sizes?

I wanted to run some tests with some RL algorithms in a continuous control task, namely PPO-clip and SAC. When comparing their NN structures described in their papers, SAC used 2 layers with 256 ...
kitaird's user avatar
  • 115
0 votes
0 answers
341 views

Is it possible to use Softmax as an activation function for actor (policy) network in TD3 or SAC Reinforcement learning algorithms?

As I understand from literature, normally, the last activation in an actor (policy) network in TD3 and SAC algorithms is a Tanh function, which is scaled by a certain limit. My action vector is ...
Bi0max's user avatar
  • 101
3 votes
0 answers
2k views

How to interpret the training loss curves in Soft-Actor-Critic (SAC)?

I am using stable-baseline3 implementation of the Soft-Actor-Critic (SAC) algorithm. The plotted training curves look promising. However, I am not fully sure how to interpret the actor and critic ...
Manuel's user avatar
  • 45
1 vote
1 answer
456 views

How to enforce action bounds between 0 & 1 in soft actor-critic algorithm?

In the paper "Soft Actor-Critic Algorithms and Applications", appendix C shows enforcing action bounds using the tanh squashing function which is in (-1, 1). I have action bounds in (0, 1), ...
Mika's user avatar
  • 341
4 votes
0 answers
3k views

Optimal episode length in reinforcement learning

I have a custom environment for stock trading where an episode can be as long as 2000-3000 steps. I've run several experiments with td3 and sac algorithms, average reward per episode flattens after ...
Mika's user avatar
  • 341
0 votes
0 answers
50 views

How is the discounted maximum entropy objective obtained for soft-q-learning and SAC

In the soft q-learning paper, they provide an expression for the maximum entropy objective that takes discounting into account. My main question is: can someone explain how they incorporated ...
quest ions's user avatar
1 vote
0 answers
324 views

How to make SAC (Soft-Actor-Critic) learn a policy?

I cannot make SAC learn a task in a certain environment. The point is that it actually sometimes finds a very good policy, but it never learns the policy in the end. I am using the SAC implementation ...
vega's user avatar
  • 11
3 votes
0 answers
584 views

In Soft Actor Critic, why is the action sampled from current policy instead of replay buffer on value function update?

While reading the original paper of Soft Actor Critic, I came across on page number 5, under equation (5) and (6) $$ J_{V}(\psi)=\mathbb{E}_{\mathbf{s}_{t} \sim \mathcal{D}}\left[\frac{1}{2}\left(V_{\...
DannyBoi's user avatar
0 votes
1 answer
1k views

Why is my Soft Actor-Critic's policy and value function losses not converging?

I'm trying to implement a soft actor-critic algorithm for financial data (stock prices), but I have trouble with losses: no matter what combination of hyper-parameters I enter, they are not converging,...
Zahra's user avatar
  • 3
3 votes
1 answer
530 views

Where does entropy enter in Soft Actor-Critic?

I am currently trying to understand SAC (Soft Actor-Critic), and I am thinking of it as a basic actor-critic with the entropy included. However, I expected the entropy to appear in the Q-function. ...
Vildemort's user avatar
  • 117
1 vote
0 answers
425 views

How does the automated temperature adjustment step work in Soft Actor-Critic?

In section 5 of the paper Soft Actor-Critic Algorithms and Applications, it is proposed an optimization problem to obtain an optimal temperature parameter $\alpha^*_t$. First, one uses the original ...
Diego Gomez's user avatar
4 votes
2 answers
590 views

What is the gradient of the objective function in the Soft Actor-Critic paper?

In the paper Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, they define the loss function for the policy network as $$ J_\pi(\phi)=\mathbb E_{s_t\...
Maybe's user avatar
  • 471