Predict next event based on previous events and discrete reward values

Question

Suppose, I have several sequences that include a series of text (the length of sequence can be varied). Also, I have some related reward value. however, the value is not continuous like the text. It has many missing values. Here is an example of the dataset.

Sequence 1        Sequence 2 .............. Sequence n
------------      ----------                -------------
Action  Reward    Action  Reward            Action  Reward
  A                 C                          D
  B       5         A                          B      6
  C                 A       7                  A       
  C       6         B       10                 D           
  A                 C                          A           
  B       2         A                          B           
  ..                ...                        ...
 ...               .....                      .....
  D       5         C      4                   D          

Now I want to predict the next action based on the reward value. The idea is I want to predict the actions that leads to more rewards. Previously, I used only action data to predict the next action using LSTM and GRU. However, how could I use this reward value in this prediction? I was if thinking Reinforcement learning (MDP) could solve the problem. However, as the rewards are discrete, I am not sure if RL could do that. Also, is it possible to solve this problem with Inverse RL? I have some knowledge of deep learning. However, I am new to reinforcement learning. If anyone gives me some suggestion or provide me useful papers link regarding this problem, it could help me a lot.

Answer 1

Your problem does look like it could be a good match to reinforcement learning, or at least the related idea of contextual bandits. Whether or not it would be a good match to the full reinforcement learning algorithm depends on whether any of the data you are processing could be considered part of an environment state, and whether or not that state evolves based on rules that an agent could learn to take advantage of.

Previously, I used only action data to predict the next action using LSTM and GRU. However, how could I use this reward value in this prediction?

There are a few different ways to do this using reinforcement learning theory. The simplest is to build a regression predictor that approximates a sum of future rewards (also known as the return or utility) depending on a proposed action from the current state. Then you could use the value function approximator (formal name for the predictor you just built) to predict results from each possible action and pick the maximising one. It is possible to learn such a value function from a historical dataset using methods such as Q learning.

The subject is too complex to teach from scratch in a single answer here. A good learning resource is Reinforcement Learning: An Introduction by Sutton & Barto, which the authors have made available for free.

However, as the rewards are discrete, I am not sure if RL could do that.

Yes it can. Reinforcement learning just requires that rewards in each situation follow a consistent distribution of values. Always returning discrete values is not a problem, neither is always returning the same value in the same situation. Randomness in the reward value - such as sometimes returning a discrete value and other times not in the same situation - is also OK. You can treat missing values as zero, since you are concerned only with the sum of received rewards, using zero when no value is available has no effect on what will be considered the optimal solution.

Also, is it possible to solve this problem with Inverse RL?

Probably not. Inverse RL is concerned with figuring out the parameters that an existing agent is working with by observing it. For instance you could use it to observe a creature's behaviour and figure out which rewards were more valuable to it. In your case you have the reward values, so you don't need to figure them out.

Caveat: You need to figure out what constitutes state in your environment. If there is some state that can be used for predictions, but the agent's behaviour never changes the state, then you may want to spend some time modelling your problem as a contextual bandit instead. Bandit algorithms are introduced in the same book, Reinforcement Learning: An Introduction, but only as much is needed to teach about the full RL problem - bandit solvers can get far more sophisticated than the book considers.

Note that if the history of agent actions impacts the reward (e.g. it is a matter of timing when to take the right action), then that history is part of the state, and you likely do have a full reinforcement learning problem to solve.