I have a time series data with a little unusual cost/reward function (I haven't seen it before)

The model must predict a $Y$ value for any $X(t)$.

The reward is computed as follows. The model will receive a reward equal to $Y_\text{true} * Y_\text{prediction}$. But if the reward is a positive value, the model won't receive a positive reward in next $5$ time steps (it will get negative rewards anytime). It means sometimes it is better for the model to predict 0 and wait for a better reward.

I have two questions:

  1. Is it a supervised learning or reinforcement learning problem?

  2. If it is a supervised learning, which optimization method should I use for it?

  • $\begingroup$ It might be a good idea to describe what $Y$ is in your case (I suppose it's a number given the way you're computing the "reward") and why the reward is computed in the way you're describing, and describe your problem more in a high-level manner, if possible, i.e. what does this time-series data represent? Moreover, please, have a look at my last edit to your post to understand how to format your next post more properly. $\endgroup$ – nbro Apr 16 at 13:44
  • $\begingroup$ Is the "prediction" really labelled as such in the original problem? It seems to me that as long as the player/agent in this game is confident in the sign of $X(t)$ it should make the magnitude of its output as high as possible and not really related to actual value of $X(t)$. Is there any statement or element of this problem that rewards $Y_{prediction}$ being close to $Y_{true}$? $\endgroup$ – Neil Slater Apr 16 at 16:44
  • $\begingroup$ Some questions which may help decide whether this is RL: Is a "prediction" expected on every timestep? Is the objective to obtain the most reward over some number of timesteps (e.g. a game or episode which ends) or on average? (If not, what exactly is the objective for this game). What form does the value at $X(t)$ take, and is it expected to contain data that can be used to infer the distribution of $Y_{true}$ in isolation (i.e. the value is not dependent on $X(t-1)$ in any way)? $\endgroup$ – Neil Slater Apr 16 at 16:51
  • $\begingroup$ @NeilSlater no this is not really a "prediction" problem.I use "Y prediction" label for model output value. Let's assume the problem as a 2d game which the agent is moving forward on a tiled ground. its x position : x = t . it can dig the ground at anytime to find a reward = R(x), the reward can be a positive value (i.e coins in a chest) or negative (some bombs) also it can be zero .if the agent hits a chest it cannot mine another chest in next 5 steps. The model objective is to decide to dig or skip the X(t) based on its last time-series features. we have all R(x) values in train data $\endgroup$ – MohamadAli Zeraatkar Apr 16 at 18:54
  • $\begingroup$ Thanks, that makes it clear. It does seem a lot like an RL problem. Unrelated though, you give the reward equation without setting any bounds on the "prediction". Can I predict a value of 10,000,000 in the hope that the value of $Y_{true}$ for $X(t)$ is +1, and gain 10,000,000 reward? Or are there constraints on what I am allowed to predict? $\endgroup$ – Neil Slater Apr 16 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.