Hello I am new to reinforcement learning and robotics. So far I have an understanding of the concept on 2D world. You can make agent move one step in one direction. However, how do you define movement action of a robot arm? I am a bit lost over here. Any useful links or keywords would be very appreciated! :)

  • $\begingroup$ What do you mean by "define movement action of a robot arm"? $\endgroup$
    – nbro
    Feb 18, 2019 at 17:01
  • $\begingroup$ in 2D grid world agent can move up, left, right, down. Hence, has option of 4 moves. But the robot arm can move in any direction in any amount. So my question was how to make agent choose an option of move and how to define them. $\endgroup$
    – user22442
    Feb 19, 2019 at 18:08

1 Answer 1


It depends a lot on the hardware of your robot arm. Assuming that your servos have encoder information, if you have access to servos that have limited control like "rotate left/rotate right" functionality, you can phrase the your action space to be ["move left", "stop", "move right"]. In this way you can implement a discrete action space with 3 actions per servo and have an agent learn to move the servos around the space.

If your servos are connected to each other in an elbow/shoulder configuration, you can have a 9 discrete action setup essentially making a box of cardinal directions:




If you have 3 or more servos, you can still use the same idea of discrete actions but the number of discrete actions grows by a factor of 3 with each servo as your action space is now the cross product of all of the other servos.

Alternatively you can use a "multi-headed" agent where each head chooses actions for a certain servo but there are pros and cons for both depending on your usecase.

If you have more advanced servos like Dynamixels which have high quality encoders, you'll have access to more advanced controls schemes. For instance, Dynamixels allow you to give actions in encoder space, angle space, and even velocity space. For example, you could give the action of "go to encoder value of 500" or "go to 90 degrees" or "move .5 radians/second". All of these approaches are useful for certain tasks. For humans controlling the arm using a joystick, the velocity based is the most intuitive and, in my experience, the same is true for RL agents using continuous control.

If you are using continuous control, you should normalize all of your action spaces within your agent then "unnormalize (?)" them before giving the actions to your servos. For instance if your servo velocity ranges from -3.5 rad/sec to +3.5 rad/sec, have your agent select actions in the range of [-1,1] then multiply by 3.5 to get the velocity.

In either case, one thing that should be noted is that you give your robotic arm enough time to actually perform the action that the agent selected. If not you will see your robot "jitter" back and forth quickly as your agent selects actions randomly. This is bad for a few reasons but most importantly because it might break your servos. To overcome this issue, give each action a little more time to be "actualized" by your robot. This can either come in the form of adding a delay in your code while your servos do "the thing" or by using an "iterations since last action update" counter to get a new action from the agent after a certain number of iterations. Not only is this better for your hardware, this also leads to better exploration of your state space as your agent can move through the state space encountering similar states more frequently.

A last thing to be aware of is to set hard coded limits on the servos so that your agent doesn't kill your servos by banging the arm against a table or something. Finding these safe bounds isn't always easy as multiple jointed configurations can have multiple ways of hitting the bounds and it might require a forward dynamics model to find these limits. If you have only 2 servos, it should be pretty easy to find though :).

  • $\begingroup$ Thank you very much for such detailed answer! We have a 6 axis robot at uni. I can tell robot to go to point where coordinates are x, y, z. Hence, I can define boundary box easily. Should I define movements of one unit in x, y, z for each possible direction? $\endgroup$
    – user22442
    Feb 19, 2019 at 18:15
  • $\begingroup$ That is definitely a possibility. It depends on the application. A 6 axis arm with coordinate controls implies that there is a lowerlevel system of forward dynamics and when given a command of "go to xyz", there are infinitely many rotations of the joints that can perform that movement and the forward dynamics model finds 1 based on some metric (distance). If the agent doesnt need to know about objects that are in its reach then I would say this is fine. If, however, the agent would need to know how to avoid hitting certain objects, then only using xyz isnt enough and youd need servo control. $\endgroup$ Feb 19, 2019 at 18:31
  • $\begingroup$ Jaden, thank you very much. Now I understand it much better. To start, collision control is not a problem but aim is to have it in action. I have an idea for that as well. Ačiū labai!!! $\endgroup$
    – user22442
    Feb 20, 2019 at 20:00

You must log in to answer this question.