1
$\begingroup$

I am working on an RL Problem that drives me nuts. My goal is to control a robot arm in a simulator that has to do 2 things:

  1. Hold the arm in a certain position (that is easy and done)
  2. If I apply an external force, I want the arm to displace a certain amount, which I have defined beforehand as the "stiffness goal".

The problems forms a 2 objective optimization problem and I handle that as a sum of weighted rewards, meaning my total reward should be $R = w_b*r_b+w_s*r_s$ with $r$ and $w$ being the rewards and weights for balancing and stiffness. However, that function is difficult to give to the agent in practice.

A reward for the stiffness can only be given at every n timesteps, because in order to do that I have to interact with the system (-> apply external force) and observe the arm's displacement for some other mtimesteps. After that I can compute the reward for this displacement with respect to my desired displacement. That is only once per episode in my training setup. On the other hand I can easily give the balancing reward at every simulation timestep. That is only based on the current position and that information is always available. But the stiffness reward is only available after I apply external force. So I have a very dense balancing reward, and a very sparse stiffness reward.

I feel like no matter what I do, the agent just cannot learn the stiffness goal.

I have tried setting $r_s$ to 0 at the timesteps where I don't have it, and tried all kinds of different weights if $r_s > 0$. I have also tried using the last measured stiffness as stiffness for the following timesteps. None of this worked.

Does anybody have any suggestions how to handle such a problem? Right now I am using PPO and an MLP network, with the standard parameters from SB3, except for the learning rate and KL Divergence which I set to $10^-5$ and $10^-4$ respectively.

Improve this question
$\endgroup$
7
  • $\begingroup$ Is it possible to reformulate the task to hold the arm in the target position modified by force/stiffness vector? That removes the separate stiffness reward, although it doesn't remove the sparseness of experience (arm still needs to apply ongoing force to maintain position) $\endgroup$
    – Neil Slater
    Commented Mar 8 at 8:27
  • $\begingroup$ You mean by sensing the external force, and then adapting to a new target position which is calculated from the desired displacement? I tried that already and it's a bit against the point, also like this the agent does not learn to swing back to it's inital position after the push is over. The balancing target position should be constant the whole time. After the push the agent should "willingly" digress from that target, make the desired swing and go back to the target position. It's my uni thesis project and definitely poses more problems than i thought. $\endgroup$
    – mavex857
    Commented Mar 8 at 10:37
  • $\begingroup$ So it's not a force (to balance statically), but an "impulse" (to react dynamically)? How is it being sensed? $\endgroup$
    – Neil Slater
    Commented Mar 8 at 10:45
  • $\begingroup$ Yes that's exactly correct. The force is only applied for 1 timestep. It's being sensed by a sensor in the simulator. So I do have the information of when that push happened and with what force. $\endgroup$
    – mavex857
    Commented Mar 8 at 12:01
  • $\begingroup$ And the force history is kept in the state, available to the agent until no longer required? Otherwise it would be non-Markov. (Or maybe your model is RNN?) $\endgroup$
    – Neil Slater
    Commented Mar 8 at 12:03

0

Reset to default

You must log in to answer this question.