Reinforcement learning with action consisting of two discrete values

Question

I'm new to reinforcement learning. I have a problem where an action is composed of an order (rod with a required length) and an item from a warehouse (an existing rod with a certain length, which will be cut to the desired length and the remainder put back to the warehouse).

I imagine my state as two lists of a defined size: orders and warehouse, and my action as an index from the first list and an index from the second list. However, I have only worked with environments where it was only possible to pick single action and I'm not sure how to deal with two indexes. I'm not sure how DQN architecture should look like to give me such action.

Can anyone validate my general idea and help me find a solution? Or maybe just point me to some papers where similar problems are described?

Answer 1

You would still be picking a single action. Your action space is now $\mathcal{A} = \mathcal{O} \times \mathcal{I}$ where I've chosen $\mathcal{O}$ to be the set of possible orders from your problem and $\mathcal{I}$ to be the set of possible items.

Provided both of these sets are finite, then you should still be able to approach this problem with DQN. Theoretically, this should be easy to see, as any element from $\mathcal{A}$ is still a single element it just happens that this element is now a tuple.

From a programming point of view, let's consider the simple example of cartpole, where the possible actions are left and right. Your $Q$-function obviously won't know the meanings of 'left' and 'right', you just assign it to an element of a vector, i.e. your $Q$-function would output a vector in $\mathbb{R}^2$ with e.g. the first element corresponding to the score for 'left' and the second element corresponding to the score for 'right'. This is still the case in your problem formulation, you will just have a $Q$-function that outputs a vector in $\mathbb{R}^d$ where $d = |\mathcal{A}|$ - you would just have to make sure you know which element corresponds to which action.

Also, there is the possibility that this approach could leave you with a large dimensional vector output, which I imagine would probably mean you'd need more simulations to properly explore the action space.

Hope this helps.