# What reinforcement learning algorithm should I use for the following problem?

Environment

I have a static timeseries environment meaning the environment is the same.
This problem is a multi armed bandit problem.

Time t0 t1 t2
State s0 s1 s2
Score 10 0.1 0.2
Class 1 0 0

Keep work = 1
Class = (Score>9)

Rewards

Reward = $$\frac{Score}{Work}$$

I have various states in the dataset which are ordered chronologically and each state contains a reward from (0 -> inf) rewards come in 2 types a very small reward then very sparsely a high reward is given.

To calculate the reward from a state is an expensive operation. The model can discard states to increase the reward density reward per work.

For example lets consider 2 cases\

• model 0: do nothing
• model 1: optimal
Model Metric t0 t1 t2
model 0 decision keep keep keep
model 0 score 10 0.1 0.2
model 0 work 1 1 1
model 1 score 10 0 0
model 1 work 1 0.1 0.1
• model 0: reward = $$\frac{10+0.1+0.2}{1+1+1}$$ = 3.43

• model 1: reward = $$\frac{10+0+0}{1+0.1+0.1}$$ = 8.33

To avoid the model discarding most states to make work value low a penalty can be applied if the model discards too many states. this is due to if keep/discard work is decreased reward will increase. As $$reward\propto\frac{1}{work}$$

States

Each timestep state is related by being an input to a process and is monotonic The state is equal to the process with the timestamp as the input Sn = P(tn)

Domain Knowledge

1. P is unknown
2. high rewards are very sparse >1:10000

Attempts so far/Ideas
I have already tried using Bayesian optimisation on this problem with minimal luck.

there are many different types of reinforcement learning algorithms. The ones applicable are

1. output is classification(category) for keep/discard continuous state space -> discrete action space

stackoverflow

• TD3
• DDPG
• SAC
• PPO
1. output is decision boundary(value) for classification continuous state space -> continuous action space

2. output is feature(value) which is then classified continuous state space -> continuous action space

• A2C
• REINFORCE
• CACLA

Questions

1. What reinforcement learning algorithm should I use?

2. given the problem is 1,2 or 3 most sensible?

3. any other considerations I should be aware of?

Thanks

• Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
– Community Bot
Mar 14 at 15:02