How to output an integer/discrete number n with a single output neuron?

Question

Say I have a game with 4 base actions [left, right, up, down] and then a value n, which determines how many times the chosen action is repeated.

For example, action = left, n = 3 -> go left 3 times. In this game $(left,1)*3 \neq (left,3)$ as negative reward is handed out at every single time step (this is for research purposes, so it cannot change).

I would like to test how a DDQN and a DQN algorithm are affected, as I increase the number of actions available (increase $n$).

My question is; Is there a smarter way to implement this, other than increasing the depth of the output layer? I.e len(output_layer) = n?

I was thinking of whether or not, there was a way for a single neuron to determine n and then have 4 other neurons that determine the best action? Would this even have any positive effects? (such as less training time, faster computation, better generalization, etc.)

If yes, how would this typically be done?

Answer 1

Predicting the correct amount of repetitions for an action sounds like a regression task. Turning it into a classification task using a model with n output nodes will lead to several drawbacks, the biggest ones being:

• Having to choose a priori a finite max amount of actions n
• Turning the data into really sparse vectors, especially for large n.

So a better choice in my opinion would be to treat this problem as a regression one, i.e. using a single output node trained on predicting continuous values, and round the predictions to convert real values to integers.

Of course the task of choosing an action is a classification one, but it is possible to combine classification and regression into a single model, using Multi Task Learning, the idea being to train a model with one head (input layer) one body (some hidden layers) and 2 tails (output layers), one trained on classification and one trained on regression. The image below is an example of such model (the t subscript in the output layers stands for task). There are several other options to accomplish the same results, so check this survey as well.

Last but not least, if multitask learning is not suitable for you, you could simply train 2 individual models, one for action selection and one for repetition prediction. If going down on this route I would suggest to perform classification first and then repetition prediction, since the output of the classification model would be relevant for the regression one.