What kind of observation state would you give for that environment?

I'm making a new environment where I have two sphere (one above the other) in a 2D plan.

I would like some advice on what observation state I should give to my RL. Today I have given the following:

1. Position of A (X)
2. Angular velocity of A
3. Position of B (X and Y)
4. Angular velocity of B
5. Angle between A and B (relative to A)

This gives in total 6 different state. The goal is to keep the sphere B as long as possible above A. I have thought a lot about the observation state and I don't know if the position of B should be relative to the position of A ? or should it be relative to the world ?

Should I remove or add other state ? WDYT ?

Edit

I forgot to mentioned crucial information on my environment: Both disc have different weight and size (as on the picture) and physics is also enabled. "A" has a weight of 1.5 Kg and "B" 0.5Kg. There is a linear damping on both disc of 0.01. It's a constant that can be adjusted in my environment.

Provided the discs do not have any properties that vary over their surfaces, and you are not trying to generalise to different sizes/weights of discs for a single agent, then your state variables seem like a complete description that you could use to predict the dynamics of the combined system and consequences of simple actions (presumably your actions will be torque on either A or B in order to maintain balance). However, I think it can be simplified without losing any information for a value or policy function.

Assuming the discs always remain in contact, I would recommend you don't track B's position in X,Y - its angular position with respect to A's centre, and A's X position define everything you need to know.

Asuming you have a simple, perfect friction version with no slipping, but constraints on position of A (similar to cartpole) then I think you have four variables:

1. Angular velocity of A ($$\omega_a$$ rotation speed)
2. Angular velocity of B ($$\omega_b$$ rotation speed)
3. Angular position of B's contact with A ($$\theta_c$$ rotation offset, measured from centre of A)*
4. Angular position of A with respect to a start position ($$\theta_a$$ rotation offset) OR position of A's centre with respect to a start position ($$x_a$$ distance offset). These are in a simple ratio $$x_a = r_a\theta_a$$ where $$r_a$$ is A's radius, so you can use either

You can drop this to 3 variables if there is no constraint on the position of disc A, because its position is not required to predict the dynamics for balancing B on top of A.

The physics details you added to the question don't appear to change any of the above. If you want to train an agent so that it can generalise to different values in the same trained model, then you would add the params you want to vary to the state. Plus you would need to train with different values so the approximator can learn how they impact action choices.

There is another possibility, that I have assumed you are not considering. If A and B's surfaces are allowed to slip - i.e. there is not perfect friction between them, then you potentially have another free variable in that B's motion around the surface of A is not simply determined by combination of A and B's angular velocity. Whether or not there is another variable needed will depend on how you calculate this slippage. Simple friction models will not need another variable, but more complex ones could require state to track whether slip is occurring.

* I put a $$c$$ suffix on this to differentiate it from the natural reading of $$\theta_b$$ which would be B's rotation offset around its own centre.

• Hi Neil! Thanks for your answer (again)! Reading your answer, I found that I missed to include crucial information about the environment and I will edit my question to include them. In fact, both disc have different weight and size (as on the picture) and physics is also enabled. "A" has a weight of 1.5 Kg and "B" 0.5Kg. There is a linear damping on both disc of 0.01. It's a constant that can be adjusted in my environment. I guess all the information I just gave modify a lot the observable state... ? Also why "A" position is necessary ? What if -inf < X < inf ? Commented Sep 16, 2023 at 18:05
• If there are no constraints on A position, then it is not necessary state for the RL agent to predict value, so you could drop it. I don't think any of the environment parameters that you commented impact the RL state unless you want to train the agent so that it can operate in variations of the environment without re-training. If you train and operate the agent with the same parameters, then those params should not change anything. I don't see any physics that would add more degrees of freedom in the state, but that is possible, and I mentioned friction and slipping models as possibility. Commented Sep 16, 2023 at 20:06
• @CyDevos I'm always happy to answer questions on the site, but I cannot realistically make time to DM and consult on projects, or commit to support a project over time. Here, I think some physics knowledge combined with RL is required to spot the differences between your simulator state and RL state, but lots of experts on this site should have both. Commented Sep 16, 2023 at 20:20
• @CyDevos Sure that makes sense for the balancing task, effectively the return is how long the agent can maintain balance. By the way, I assumed in my answer that an episode would end if the discs stopped touching - similar to cartpole, the game would give up if you've obviously lost control. That's a guess on my part - if I'm wrong about that, the answer could also be wrong Commented Sep 17, 2023 at 8:33
• as promise, just to let you know that I have successfully make my ball A keep the ball B on top of itself! Commented Sep 23, 2023 at 10:15