# Is this a supervised or reinforcement learning problem, and which algorithm should I use to solve it?

I have a time series data with a little unusual cost/reward function (I haven't seen it before)

The model must predict a $$Y$$ value for any $$X(t)$$.

The reward is computed as follows. The model will receive a reward equal to $$Y_\text{true} * Y_\text{prediction}$$. But if the reward is a positive value, the model won't receive a positive reward in next $$5$$ time steps (it will get negative rewards anytime). It means sometimes it is better for the model to predict 0 and wait for a better reward.

I have two questions:

1. Is it a supervised learning or reinforcement learning problem?

2. If it is a supervised learning, which optimization method should I use for it?

• It might be a good idea to describe what $Y$ is in your case (I suppose it's a number given the way you're computing the "reward") and why the reward is computed in the way you're describing, and describe your problem more in a high-level manner, if possible, i.e. what does this time-series data represent? Moreover, please, have a look at my last edit to your post to understand how to format your next post more properly. – nbro Apr 16 at 13:44
• Is the "prediction" really labelled as such in the original problem? It seems to me that as long as the player/agent in this game is confident in the sign of $X(t)$ it should make the magnitude of its output as high as possible and not really related to actual value of $X(t)$. Is there any statement or element of this problem that rewards $Y_{prediction}$ being close to $Y_{true}$? – Neil Slater Apr 16 at 16:44
• Some questions which may help decide whether this is RL: Is a "prediction" expected on every timestep? Is the objective to obtain the most reward over some number of timesteps (e.g. a game or episode which ends) or on average? (If not, what exactly is the objective for this game). What form does the value at $X(t)$ take, and is it expected to contain data that can be used to infer the distribution of $Y_{true}$ in isolation (i.e. the value is not dependent on $X(t-1)$ in any way)? – Neil Slater Apr 16 at 16:51
• @NeilSlater no this is not really a "prediction" problem.I use "Y prediction" label for model output value. Let's assume the problem as a 2d game which the agent is moving forward on a tiled ground. its x position : x = t . it can dig the ground at anytime to find a reward = R(x), the reward can be a positive value (i.e coins in a chest) or negative (some bombs) also it can be zero .if the agent hits a chest it cannot mine another chest in next 5 steps. The model objective is to decide to dig or skip the X(t) based on its last time-series features. we have all R(x) values in train data – MohamadAli Zeraatkar Apr 16 at 18:54
• Thanks, that makes it clear. It does seem a lot like an RL problem. Unrelated though, you give the reward equation without setting any bounds on the "prediction". Can I predict a value of 10,000,000 in the hope that the value of $Y_{true}$ for $X(t)$ is +1, and gain 10,000,000 reward? Or are there constraints on what I am allowed to predict? – Neil Slater Apr 16 at 19:08