Questions tagged [soft-actor-critic]

For questions about Soft Actor-Critic (SAC), which was proposed in "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor" by Tuomas Haarnoja et al.

6 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
0answers
143 views

In Soft Actor Critic, why is the action sampled from current policy instead of replay buffer on value function update?

While reading the original paper of Soft Actor Critic, I came across on page number 5, under equation (5) and (6) $$ J_{V}(\psi)=\mathbb{E}_{\mathbf{s}_{t} \sim \mathcal{D}}\left[\frac{1}{2}\left(V_{\...
1
vote
0answers
41 views

How to make SAC (Soft-Actor-Critic) learn a policy?

I cannot make SAC learn a task in a certain environment. The point is that it actually sometimes finds a very good policy, but it never learns the policy in the end. I am using the SAC implementation ...
1
vote
0answers
197 views

How does the automated temperature adjustment step work in Soft Actor-Critic?

In section 5 of the paper Soft Actor-Critic Algorithms and Applications, it is proposed an optimization problem to obtain an optimal temperature parameter $\alpha^*_t$. First, one uses the original ...
0
votes
0answers
26 views

How to interpret the training loss curves in Soft-Actor-Critic (SAC)?

I am using stable-baseline3 implementation of the Soft-Actor-Critic (SAC) algorithm. The plotted training curves look promising. However, I am not fully sure how to interpret the actor and critic ...
0
votes
0answers
64 views

Optimal episode length in reinforcement learning

I have a custom environment for stock trading where an episode can be as long as 2000-3000 steps. I've run several experiments with td3 and sac algorithms, average reward per episode flattens after ...
0
votes
0answers
24 views

How is the discounted maximum entropy objective obtained for soft-q-learning and SAC

In the soft q-learning paper, they provide an expression for the maximum entropy objective that takes discounting into account. My main question is: can someone explain how they incorporated ...