Are there limitations on network output architecture and action mapping in reinforcement learning?

Question

I'm easing my way into a toy reinforcement learning problem where my objects can move and take different actions on a simple grid, but I'm having trouble understanding what constraints might exist in how I build my output layer and map it to the environent. I understand that, since vanilla backpropagation doesn't make sense in this context, we use methods like Policy Updating or Q-Learning to be able to differentiate with respect to the gradient. However, it is unclear to me if these methods impose constraints on the "form" of the output layer and its mappings to actions?

For instance, in the examples I read, typically there is one output node for each of the available actions in the environment. However, in a complex environment I could see one wishing for a variety of output forms. Perhaps, in my case, I desire to have one series of nodes determine the action to take and another node (i.e. a sin output for the angle) or two (i.e. determining the x/y) to determine the direction (i.e. "grab" action at "angle/position"). Is this viable, and indeed are any arbitrary mappings from the output layer to the environment valid? Or does current reinforcement learning techniques constrain the output and mapping (such as each node must be a possible action choice)?

Answer 1

Answering my own question due to little traction here--- the answer is yes, there are constraints, often of the form of one output node for each action.

In order to get a gradient differentiable with the network, the network generally must output one node for each of the available actions. Broadly speaking, reinforcement learning will take the perspective of either Q-learning or Policy Gradients as the algorithm used to update the network, both of these effecting the structure and interpretation of the output. In Q-learning, each output node will essentially represent the q-value or expected value return on that action. In the Policy Gradient approach, this output layer is generally capped by a softmax function essentially converting each node to the probability of an action, making the output layer a probability distribution over the available actions. The actual action taken in this case is through sampling from that distribution (so there can be some stochasticity). There is also the possibility of flipping this idea of input = state, output = action around in some architectures so that the input is the state after a given action, and the output is a single node (representing, for instance, the q-value).

However, in either case the network output is indeed constrained in order match the design needs created by the overall algorithm type (i.e. q-learning or policy gradient or similar such flavors).