0
$\begingroup$

To be able to explain my question I thought it is probably better to consider the following example: Let's take an environment, where a bridge crane need to lift a barrel from the position "start" and move it to the position "goal" moving along the axes: $X, Y, Z$.

example

But the movement cannot be straight to the goal. The movement from the start to the goal is based on the presence of obstable in the enviroment and some pratical rules (e.g. the barrel must be lifted first, then horizontally moved and at the end dropped at the goal position).

I'm struggling to figure out, how to train an agent to be able to accomplish such a task. But here I want to share my thoughts:

  1. At first I thought to use PPO or SAC as model free algorithms and assigning different rewards, when reached the goal and the positions in between (let's call them waypoints) In other words, the agent starts from the start position and gets a reward of +100 points if the agent reaches $a$ in some time. Now... in order to move forward I would need to give a higher reward to $b$. But that means, that the agent will try during the training to reach $b$ directly instead going to $b$ over $a$.
  2. Even if the issues in the first point are somehow solvable there is the problem, that the agent would learn to reach the goal through the waypoints seen during the training, even if the new environment is now different (obstacles are in different position, the waypoints changed their positions, etc.). So the agent would learn "memorizing" the training environment, without generalization.

What could be the best strategy in such enviroments? Thanks

Improve this question
$\endgroup$
4
  • $\begingroup$ Have you tried to combine PPO/SAC (or similar) with Hindsight Experience Replay? HER allows to train goal-based policies. Maybe you can split the problem in subgoals, having also a goal-dependent reward function. For the generalization issue you may need to apply domain randomization and/or extend the state representation to include information about the obstacles in the environment. Also,I think that randomizing the start/goal positions as well the obstacles during training should help $\endgroup$
    – Luca Anzalone
    Commented Jul 19 at 16:07
  • 1
    $\begingroup$ There are many things to unpack here: 1) PPO is onpolicy HER is for offpolicy algorihtms. So it could fit at best with SAC 2) Yes I tried SAC + HER: it miserably fails. The reason is that in case of HER I need a sparse reward (e.g. +1 if the (sub) goal is reached, 0 otherwise). Since the parameter space is continuous there are too many states to be explored, it simply does not work. For this reason I switched back to shaping the reward instead using a sparsed one $\endgroup$
    – Dave
    Commented Jul 20 at 8:50
  • $\begingroup$ 3) You say. "split the problem in subgoals". But this is actually the point. How? Should I train the agent to move first up. And when it does it, it should learns to move horizontally? 4) Domain randomization is what I'm doing now. The result is that the agent does not converge to a solution $\endgroup$
    – Dave
    Commented Jul 20 at 8:53
  • $\begingroup$ Maybe you can try to train multiple policies (agents), and then coordinate them by hand-made rules: you can have an agent that lifts the barrel up, then another specialized for moving it horizontally, and eventually another than puts the barrel on the ground. Maybe this can help. Also including the goal's coordinate in your states may help, or have goal-conditioned value functions. In addition, make sure your reward shaping makes the environment actually solvable! $\endgroup$
    – Luca Anzalone
    Commented Jul 22 at 7:27

1 Answer 1

Reset to default
2
$\begingroup$

I do not have enough reputation to just comment, but here are my thoughts:

Do you actually need to use subgoals/waypoints? If your objective is simply how to train an agent to be able to accomplish such a task, then I am thinking you do not need to break it into subgoals/waypoints. Indeed, the environment you describe can be reduced to a 2d environment with a starting position and a single goal position, where the agent must find the shortest path between these two positions. Since the barrel must be lifted first whatever happens, this can be hard coded in the reset point (I don't think there is much use in training a RL agent to learn to just lift an object as there are not many ways this can be done!), same for the drop at the goal position (simply set the routine for the drop once the agent has reached the goal position).

If you actually need waypoints, then as @Luca Anzalone suggested in the comments, you can still include the goal's coordinate in your states.

For Domain Randomization, if the agent does not observe the positions (and size?) of the obstacles, then the policy will not be able to know where it can move (this is due to the overly-cautious or "conservative" nature of DR policies). The only way this would work is if your obstacles are distributed such that there is one specific path that is always opened. Otherwise, you should consider adding the obstacle information to your states (the agent does not need to observe the full space, but it should be able to at least observe when it is close to an obstacle, so that it can avoid it or move away).

If you really need to train the policy for lifting and dropping, then you could take a look at continual reinforcement learning (specifically Multi-task Learning?), but I think that even in this case, a properly designed reward function could be enough (one that penalizes the agent for starting to move while the barrel is still on the ground [of course this means a 3d environment instead of the 2d case I mentioned above] - the penalty can be proportional to the height of the barrel and maybe to the distance from the starting or goal position).

Improve this answer
$\endgroup$
3
  • $\begingroup$ Many thanks to your interesting answer/comment. But the problem is that, as you pointed out, the reward function will be a sum of many different factors, which in worst case work against each other. So the barrel could move perfectly on the vertical but getting a penalty because it moves far away from the final goal (e.g. the goal is at the same level as the start position) $\endgroup$
    – Dave
    Commented Jul 25 at 9:26
  • $\begingroup$ @Dave Yes, reward shaping is well known problem in RL... There isn't many ways to solve the issue. Have you considered mutli-task reinforcement learning? Another possibility for training a single policy is multi-objective reinforcement learning in which the task reward and penalty are weighted by some preference vector. In particular, take a look at dynamics weight MORL algorithms: link. The paper is for Q-learning, but you can adapt it to SAC or PPO by simply using universal policy networks. $\endgroup$
    – Ahnel
    Commented Jul 26 at 3:20
  • $\begingroup$ Your intuition is correct man. But I still struggle in understanding the concept. That a scalar reward too less for complex robot system is, is clear. I read the paper carefully and looking for some simple tutorial on its implementation. But yeah, that's probably the way to go $\endgroup$
    – Dave
    Commented Jul 27 at 7:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .