I have a static timeseries environment meaning the environment is the same.
This problem is a multi armed bandit problem.

Time t0 t1 t2
State s0 s1 s2
Score 10 0.1 0.2
Class 1 0 0

Keep work = 1
Discard work = 0.1
Class = (Score>9)


Reward = $\frac{Score}{Work}$

I have various states in the dataset which are ordered chronologically and each state contains a reward from (0 -> inf) rewards come in 2 types a very small reward then very sparsely a high reward is given.

To calculate the reward from a state is an expensive operation. The model can discard states to increase the reward density reward per work.

For example lets consider 2 cases\

  • model 0: do nothing
  • model 1: optimal
Model Metric t0 t1 t2
model 0 decision keep keep keep
model 0 score 10 0.1 0.2
model 0 work 1 1 1
model 1 decision keep discard discard
model 1 score 10 0 0
model 1 work 1 0.1 0.1
  • model 0: reward = $\frac{10+0.1+0.2}{1+1+1}$ = 3.43

  • model 1: reward = $\frac{10+0+0}{1+0.1+0.1}$ = 8.33

To avoid the model discarding most states to make work value low a penalty can be applied if the model discards too many states. this is due to if keep/discard work is decreased reward will increase. As $reward\propto\frac{1}{work}$


Each timestep state is related by being an input to a process and is monotonic The state is equal to the process with the timestamp as the input Sn = P(tn)

Domain Knowledge

  1. P is unknown
  2. high rewards are very sparse >1:10000

Attempts so far/Ideas
I have already tried using Bayesian optimisation on this problem with minimal luck.

there are many different types of reinforcement learning algorithms. The ones applicable are

  1. output is classification(category) for keep/discard continuous state space -> discrete action space


  • TD3
  • DDPG
  • SAC
  • PPO
  1. output is decision boundary(value) for classification continuous state space -> continuous action space

  2. output is feature(value) which is then classified continuous state space -> continuous action space

stackoverflow CACLA paper

  • A2C


  1. What reinforcement learning algorithm should I use?

  2. given the problem is 1,2 or 3 most sensible?

  3. any other considerations I should be aware of?


  • $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Community Bot
    Mar 14 at 15:02

1 Answer 1


If your problem is a multi-armed bandit then the algorithms that you listed are not suited for the job. Those algorithms are for situations where the agent's actions has an influence on the subsequent states encountered. You should look for solutions designed for multi-armed bandit scenarios such as LinUCB.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .