People who code: we want your input. Take the Survey

Questions tagged [soft-actor-critic]

For questions about Soft Actor-Critic (SAC), which was proposed in "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor" by Tuomas Haarnoja et al.

Filter by
Sorted by
Tagged with
0
votes
0answers
34 views

Optimal episode length in reinforcement learning

I have a custom environment for stock trading where an episode can be as long as 2000-3000 steps. I've run several experiments with td3 and sac algorithms, average reward per episode flattens after ...
0
votes
0answers
24 views

How is the discounted maximum entropy objective obtained for soft-q-learning and SAC

In the soft q-learning paper, they provide an expression for the maximum entropy objective that takes discounting into account. My main question is: can someone explain how they incorporated ...
1
vote
0answers
38 views

How to make SAC (Soft-Actor-Critic) learn a policy?

I cannot make SAC learn a task in a certain environment. The point is that it actually sometimes finds a very good policy, but it never learns the policy in the end. I am using the SAC implementation ...
3
votes
0answers
125 views

In Soft Actor Critic, why is the action sampled from current policy instead of replay buffer on value function update?

While reading the original paper of Soft Actor Critic, I came across on page number 5, under equation (5) and (6) $$ J_{V}(\psi)=\mathbb{E}_{\mathbf{s}_{t} \sim \mathcal{D}}\left[\frac{1}{2}\left(V_{\...
0
votes
1answer
266 views

Why is my Soft Actor-Critic's policy and value function losses not converging?

I'm trying to implement a soft actor-critic algorithm for financial data (stock prices), but I have trouble with losses: no matter what combination of hyper-parameters I enter, they are not converging,...
3
votes
1answer
191 views

Where does entropy enter in Soft Actor-Critic?

I am currently trying to understand SAC (Soft Actor-Critic), and I am thinking of it as a basic actor-critic with the entropy included. However, I expected the entropy to appear in the Q-function. ...
1
vote
0answers
187 views

How does the automated temperature adjustment step work in Soft Actor-Critic?

In section 5 of the paper Soft Actor-Critic Algorithms and Applications, it is proposed an optimization problem to obtain an optimal temperature parameter $\alpha^*_t$. First, one uses the original ...
2
votes
2answers
216 views

What is the gradient of the objective function in the Soft Actor-Critic paper?

In the paper Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, they define the loss function for the policy network as $$ J_\pi(\phi)=\mathbb E_{s_t\...