2022 Developer Survey is open! Take survey.

Questions tagged [soft-actor-critic]

For questions about Soft Actor-Critic (SAC), which was proposed in "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor" by Tuomas Haarnoja et al.

Filter by
Sorted by
Tagged with
1 vote
0 answers
12 views

Why exclude the first entropy bonus term in the soft Q-function in SAC?

Based on OpenAI Spinning Up description of Soft Actor Critic (SAC) the soft Q-function is defined as and as they say Q value is changed to include the entropy bonuses from every timestep except the ...
user avatar
  • 11
0 votes
0 answers
16 views

Can off-policy algorithms benefit from the parallelization?

On-policy algorithms, such as A2C, A3C and PPO, leverage massive parallelization to achieve state of the art results. However, I’ve never come across parallelization efforts when it comes to the off-...
user avatar
  • 167
1 vote
0 answers
27 views

How to compare RL algorithms with different NN sizes?

I wanted to run some tests with some RL algorithms in a continuous control task, namely PPO-clip and SAC. When comparing their NN structures described in their papers, SAC used 2 layers with 256 ...
user avatar
  • 85
0 votes
0 answers
40 views

Should you clip Q values if they start to grow indefinitely?

I am training the SAC algorithm for an environment where the rewards are small as shown below and the episode length is 84. I have a problem with the Q values that grow with each step. The following ...
user avatar
  • 167
0 votes
0 answers
22 views

Relationship between standard RL and entropy regularized RL (soft Q learning)

Use the standard RL setting, denote the reward as $r(s,a,s')$, and the optimal Q function as $Q^*(s,a)$, optimal value function $V^*(s)$ and optimal policy $\pi^*(a|s) = \arg \max_a Q^*(s,a)$. In the ...
user avatar
  • 123
0 votes
0 answers
87 views

Is it possible to use Softmax as an activation function for actor (policy) network in TD3 or SAC Reinforcement learning algorithms?

As I understand from literature, normally, the last activation in an actor (policy) network in TD3 and SAC algorithms is a Tanh function, which is scaled by a certain limit. My action vector is ...
user avatar
  • 1
0 votes
0 answers
39 views

RL agent policy performs worse than random policy

I am training a trading bot with TD3 and SAC algorithms. During the first 10k steps it takes uniformly random actions before running policy learnt so far. The agent starts to do gradient descent ...
user avatar
  • 167
2 votes
0 answers
286 views

How to interpret the training loss curves in Soft-Actor-Critic (SAC)?

I am using stable-baseline3 implementation of the Soft-Actor-Critic (SAC) algorithm. The plotted training curves look promising. However, I am not fully sure how to interpret the actor and critic ...
user avatar
  • 35
1 vote
1 answer
118 views

How to enforce action bounds between 0 & 1 in soft actor-critic algorithm?

In the paper "Soft Actor-Critic Algorithms and Applications", appendix C shows enforcing action bounds using the tanh squashing function which is in (-1, 1). I have action bounds in (0, 1), ...
user avatar
  • 167
2 votes
0 answers
645 views

Optimal episode length in reinforcement learning

I have a custom environment for stock trading where an episode can be as long as 2000-3000 steps. I've run several experiments with td3 and sac algorithms, average reward per episode flattens after ...
user avatar
  • 167
0 votes
0 answers
31 views

How is the discounted maximum entropy objective obtained for soft-q-learning and SAC

In the soft q-learning paper, they provide an expression for the maximum entropy objective that takes discounting into account. My main question is: can someone explain how they incorporated ...
user avatar
1 vote
0 answers
104 views

How to make SAC (Soft-Actor-Critic) learn a policy?

I cannot make SAC learn a task in a certain environment. The point is that it actually sometimes finds a very good policy, but it never learns the policy in the end. I am using the SAC implementation ...
user avatar
  • 11
3 votes
0 answers
231 views

In Soft Actor Critic, why is the action sampled from current policy instead of replay buffer on value function update?

While reading the original paper of Soft Actor Critic, I came across on page number 5, under equation (5) and (6) $$ J_{V}(\psi)=\mathbb{E}_{\mathbf{s}_{t} \sim \mathcal{D}}\left[\frac{1}{2}\left(V_{\...
user avatar
0 votes
1 answer
526 views

Why is my Soft Actor-Critic's policy and value function losses not converging?

I'm trying to implement a soft actor-critic algorithm for financial data (stock prices), but I have trouble with losses: no matter what combination of hyper-parameters I enter, they are not converging,...
user avatar
  • 3
3 votes
1 answer
303 views

Where does entropy enter in Soft Actor-Critic?

I am currently trying to understand SAC (Soft Actor-Critic), and I am thinking of it as a basic actor-critic with the entropy included. However, I expected the entropy to appear in the Q-function. ...
user avatar
  • 117
1 vote
0 answers
263 views

How does the automated temperature adjustment step work in Soft Actor-Critic?

In section 5 of the paper Soft Actor-Critic Algorithms and Applications, it is proposed an optimization problem to obtain an optimal temperature parameter $\alpha^*_t$. First, one uses the original ...
user avatar
3 votes
2 answers
305 views

What is the gradient of the objective function in the Soft Actor-Critic paper?

In the paper Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, they define the loss function for the policy network as $$ J_\pi(\phi)=\mathbb E_{s_t\...
user avatar
  • 431