Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity

Question

I'm working on a RL problem with the following properties:

1. The rewards are extremely sparse i.e. all rewards are 0 except the terminal non-zero reward. Ideally I would not use any reward engineering as that would lead to a different optimization problem.
2. Actions are continuous. Discretization should not be used.
3. The amount of stochasticity in the environment is very high i.e. for a fixed deterministic policy the variance of returns is very high.

More specifically, the RL agent represents the investor, the terminal reward represents the utility of the terminal wealth (hence the sparsity), actions represent portfolio positions (hence the continuity) and the environment represents the financial market (hence the high stochasticity).

I've been trying to use DDPG with a set of "commonly used" hyperparameters (as I have no idea have to tune them besides experimenting which lasts too long) but so far (after 10000 episodes) it seems that nothing is happening.

My questions are the following:

1. Given the nature of the problem I'm trying to solve (sparse rewards, continuous actions, stochasticity) is there a particular (D)RL algorithm that would lend itself well to it?
2. How likely is it that DDPG simply won't converge to a reasonable solution (due to the peculiarities of the problem itself) no matter what set of hyperparameters I choose?

Answer 1

(1) You might want look into RND (Random network distillation) which allows usage of a curiosity-based exploration bonus for the agent as an intrinsic reward. You can use the intrinsic reward to complement the sparse extrinsic reward return by the environment.

The general idea is to have a randomly initialized fixed target network which encodes the next state & a predictor network is trained to predict the output of the target network. The prediction error is used to "quantify the novelty of new experience". Stronger novelty will be a good indication for the agent that it maybe worthwhile to explore more.

The authors of this (A) paper were able to achieve SOTA performance in Montezuma's Revenge, which is notorious for it's sparse reward.

In appendix A.1, It is mentioned that: "An exploration bonus can be used with any RL algorithm by modifying the rewards used to train the model (i.e., rt = it + et)." It is also mentioned that the authors combined this exploration bonus with PPO (which also works in continuous action space). In A.2, a pseudo code is provided.

I would also recommend this (B) paper (see section 3) if you're interested in exploring the available Bonus-Based Exploration Methods out there which may help in tackling hard exploration games with sparse rewards.

With regards to high stochasticity & variance, I found an interesting remark (on page 3, under Figure 2) in this (C) paper:

"our investigation of DDPG on different network configurations shows that for the Hopper environment, DDPG is quite unstable no matter the network architecture. This can be attributed partially to the high variance of DDPG itself, but also to the increased stochasticity of the Hopper task."

The remark was made in the context where the authors were trying to "tune DDPG to reproduce results from other works even when using their reported hyper-parameter settings".

Have a look here for a different benchmark on how DDPG fair against other algorithms.

(2) From the information provided, I can't conclusively provide you a quantitative assessment on DDPG performance for your specific problem. However, I would recommend the following:

(a) I will encourage you to try different RL algorithms when face with a difficult problem so that you can benchmark & find out which is more suitable. Also in (A), the authors mentioned, "PPO is a policy gradient method that we have found to require little tuning for good performance."

(b) Try different sets of hyperparameters. There are many ways to tune them systematically but discussion about this will be out of scope for this question.