What reinforcement learning algorithm should I use for the following problem?

Question

Environment

I have a static timeseries environment meaning the environment is the same.
This problem is a multi armed bandit problem.

Time	t0	t1	t2
State	s0	s1	s2
Score	10	0.1	0.2
Class	1	0	0

Keep work = 1
Discard work = 0.1
Class = (Score>9)

Rewards

Reward = $\frac{Score}{Work}$

I have various states in the dataset which are ordered chronologically and each state contains a reward from (0 -> inf) rewards come in 2 types a very small reward then very sparsely a high reward is given.

To calculate the reward from a state is an expensive operation. The model can discard states to increase the reward density reward per work.

For example lets consider 2 cases\

• model 0: do nothing
• model 1: optimal

Model	Metric	t0	t1	t2
model 0	decision	keep	keep	keep
model 0	score	10	0.1	0.2
model 0	work	1	1	1
model 1	decision	keep	discard	discard
model 1	score	10	0	0
model 1	work	1	0.1	0.1

• model 0: reward = $\frac{10+0.1+0.2}{1+1+1}$ = 3.43

• model 1: reward = $\frac{10+0+0}{1+0.1+0.1}$ = 8.33

To avoid the model discarding most states to make work value low a penalty can be applied if the model discards too many states. this is due to if keep/discard work is decreased reward will increase. As $reward\propto\frac{1}{work}$

States

Each timestep state is related by being an input to a process and is monotonic The state is equal to the process with the timestamp as the input Sn = P(tn)

Domain Knowledge

1. P is unknown
2. high rewards are very sparse >1:10000

Attempts so far/Ideas
I have already tried using Bayesian optimisation on this problem with minimal luck.

there are many different types of reinforcement learning algorithms. The ones applicable are

1. output is classification(category) for keep/discard continuous state space -> discrete action space

stackoverflow

• TD3
• DDPG
• SAC
• PPO

2. output is decision boundary(value) for classification continuous state space -> continuous action space

3. output is feature(value) which is then classified continuous state space -> continuous action space

stackoverflow CACLA paper

• A2C
• REINFORCE
• CACLA

Questions

1. What reinforcement learning algorithm should I use?

2. given the problem is 1,2 or 3 most sensible?

3. any other considerations I should be aware of?

Thanks

Answer 1

If your problem is a multi-armed bandit then the algorithms that you listed are not suited for the job. Those algorithms are for situations where the agent's actions has an influence on the subsequent states encountered. You should look for solutions designed for multi-armed bandit scenarios such as LinUCB.