1
$\begingroup$

I have a case where my state consists of relatively large number of features, e.g. 50, whereas my action size is 1. I wonder whether my state features dominate the action in my critic network. I believe in theory eventually it shouldn't matter but considering the sequential nature of RL training I am afraid the state features outweigh the action and its effect will be negligible.

What I already tried is the following:

enter image description here

Here where the state output and action are combined I use tanh activation because my action is in [-1, 1]. This led to almost flat performance from the very beginning with no improvement at all. I understand this might be due to vanishing gradients caused by tanh. I also tried the linear activation instead of the tanh, this time average episode return was fluctuating around some value with no signs of learning.

What I am currently testing is stacking the action, say 50 times, to match the number of the state features.

Any other ideas on how to tackle this issue.

Improve this question
$\endgroup$
4
  • $\begingroup$ Hi @Mika! Just to clarify - what is the input to your critic network? I.e., are you using the action as an input at all? It seems that way based on the wording of this question. $\endgroup$
    – DeepQZero
    Commented Mar 2, 2023 at 22:54
  • $\begingroup$ Hey @DeepQZero. Yes I am using the action as an input along with the state. But like i said state has a dimension of 50 and action has only one. Let me know if that helped. $\endgroup$
    – Mika
    Commented Mar 2, 2023 at 23:36
  • $\begingroup$ It helped, but I personally would need to understand more details before supplying an answer. It looks like there is an answer on this question already, and I think I'll pass on answering this question for now. $\endgroup$
    – DeepQZero
    Commented Mar 3, 2023 at 16:30
  • 1
    $\begingroup$ @DeepQZero, I added more information if that helps understand the problem better $\endgroup$
    – Mika
    Commented Mar 5, 2023 at 3:10

1 Answer 1

Reset to default
1
$\begingroup$

It is certainly possible for the state features to dominate the action features in the critic.

There are several strategies you can use:

  1. Replace the action features with a high dimensional learned embedding vector. This way you can scale up it's importance.

  2. Introduce the action at a deeper stage of the network. This way, the action is being combined with the state after the state has been compressed by the earlier layers

  3. Simply have many action features inputs that are all identical. This is similar to method 1 but easier.

I have used all 3 of these methods. They all worked well in different cases.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Hi @chessprogrammer. Thanks for the answer, I have also thought about similar methods. I tried method 2 already, i.e. learning the state embedding by reducing it to a size one representation and combining it with the action. This didn't work, basically my performance was fluctuating around some value and not improving at all. Regarding method 3 I believe you mean just to stack the copies of the action, say 50 times to match the number of the features in the state, right? I am already doing this after failing with method 2, waiting for the results, will post how it did. $\endgroup$
    – Mika
    Commented Mar 2, 2023 at 23:44
  • $\begingroup$ Not sure about the method 1, though. You mean to pass the action with size of one through several layers, then combine the learned representation with the state features, right? $\endgroup$
    – Mika
    Commented Mar 2, 2023 at 23:48
  • $\begingroup$ Hi. First off, you misunderstood method 2. I am not saying to compress the state space to one before combing. That is way too low. I am saying to make it smaller. Not that small. And method 1 I mean passing it through an embedding layer. Look up torch.nn.embedding $\endgroup$
    – chessprogrammer
    Commented Mar 3, 2023 at 0:09
  • $\begingroup$ I added a figure of what I did that I thought is similar to method 2; now I am confused. My understanding of torch.nn.embedding is that it's just a trainable look-up table; moreover, my action is continuous, not sure how I'd encode my action space. $\endgroup$
    – Mika
    Commented Mar 5, 2023 at 3:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .