Which representation is most biologically plausible for actor nodes? For example, actions represented across several output nodes which may be either

  1. mutually exclusive with each other (e.g., go north, go south, etc), achieved by winner-takes-all.

  2. NOT mutually exclusive with each other (e.g. left leg forward, right leg forward); these actions may occur concurrently. To go north, the correct combination of nodes must be active.

Similarly which representation is most plausible for critic output nodes?

  1. A single output node that outputs a real number representing the reward.

  2. A set of output nodes each representing a separate value, achieved by winner-takes-all.

Or do other representations better align with real brains ?

  • $\begingroup$ To make it clear, you want to represent both the actor and the critic, in the actor-critic architecture, as a neural network. However, it is not clear to me what your actual problem is. I understand you want to be consistent with the biology (if possible), but what exactly must be consistent with the biology, in this context? What nodes of the last layer of the neural network (both for the actor and the critic)? $\endgroup$
    – nbro
    Feb 14 '19 at 11:43
  • $\begingroup$ Hi @nbro, To be biologically plausible the model should really apply to all problems, rather than a specific problem. Some interesting example.. This model attempts to tie in with brain regions, however it uses a winner-takes-all rule for the action. This model uses plausible local learning rules for neurons, but uses a winner-takes-all rule for the action and I think requires a separate critic neuron for each state. $\endgroup$
    – thward
    Feb 16 '19 at 3:29

For the actor, I'd say the 'not mutually exclusive' option is more biologically plausible in the context of muscle systems, where the actions can be seen as simultaneous muscle activations. Maybe at a higher level, an agent thinks of the action as 'go north' or 'go south', but the final outputs which have to control muscles at a lower level have to represent simultaneous muscle activations.

For the critic, I'd say the 'single output node' is more biologically plausible. Agents perceive the world in the form of high dimensional inputs, such as images. The approach where a value function is learned in a tabular fashion where you know the value for every single state doesn't really scale very well and is limited to small discrete state spaces. For biological agents, it makes sense to have a function that senses the current state of the environment and outputs a single number that represents the value, which gives the agent an idea of how things are going so far given the actions it took in the past.

  • $\begingroup$ Hi @yewang, Yes I agree, in fact the mutually exclusive actor (1) will also not scale well in my opinion. -Do you know of any models with plausible (eg local) learning rules that support the plausible representations (2&3) ? thanks $\endgroup$
    – thward
    Feb 16 '19 at 1:17
  • $\begingroup$ For example, if you use A2C to control a robot, a common approach is to train a neural network that outputs a probability distribution over a number of actuators, often as a multivariate gaussian distribution, where the actor outputs the mean of the multivariate distribution (compatible with (2)). For the value function, it's common to model it as a function that takes an n-dimensional vector (for example joint angles) and outputs a single number representing the value of the state (compatible with (3)). Since the state space is continuous, you can't store a value for every state in a table. $\endgroup$
    – yewang
    Feb 16 '19 at 5:35
  • $\begingroup$ Hi @yewang, thanks. The examples I looked at for A2C I think are using back-propagation to train the value function. - Do you know of a paper that achieves this using only local rules ? $\endgroup$
    – thward
    Feb 17 '19 at 6:03
  • $\begingroup$ what do you mean by local rules? I think backpropagation can be thought of as local rules if you consider that the update rule for a neuron depends only on immediately close neurons. As you propagate the gradients layer after layer you get a global behavior in an indirect way, but the basic rules are local. $\endgroup$
    – yewang
    Feb 17 '19 at 9:13
  • $\begingroup$ Hi @yewang, I appreciate your feedback :) By local rules I meant it only uses local information available to each synapse. Principally hebbian type rules, including 3-factor rules (pre, post & neuromodulator). So in the case of A2C I believe the actor could be implemented with a local 3-factor rule (eg a Policy Gradient method such as ARP). But the critic value function appears to use backprop. I have not found a hebbian type rule to support that single node representation.FYI However I happen to have devised a simple method to do this, I can give you the details if you're interested. $\endgroup$
    – thward
    Feb 19 '19 at 8:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.