# Robot Arm Deep Q Learning Actions

Hello I am new to reinforcement learning and robotics. So far I have an understanding of the concept on 2D world. You can make agent move one step in one direction. However, how do you define movement action of a robot arm? I am a bit lost over here. Any useful links or keywords would be very appreciated! :)

• What do you mean by "define movement action of a robot arm"?
– nbro
Feb 18, 2019 at 17:01
• in 2D grid world agent can move up, left, right, down. Hence, has option of 4 moves. But the robot arm can move in any direction in any amount. So my question was how to make agent choose an option of move and how to define them. Feb 19, 2019 at 18:08

It depends a lot on the hardware of your robot arm. Assuming that your servos have encoder information, if you have access to servos that have limited control like "rotate left/rotate right" functionality, you can phrase the your action space to be ["move left", "stop", "move right"]. In this way you can implement a discrete action space with 3 actions per servo and have an agent learn to move the servos around the space.

If your servos are connected to each other in an elbow/shoulder configuration, you can have a 9 discrete action setup essentially making a box of cardinal directions:

Up+Left----------Up-------Up+Right

Left--------------Stop---------Right

Down+Left-----Down-----Down+Right

If you have 3 or more servos, you can still use the same idea of discrete actions but the number of discrete actions grows by a factor of 3 with each servo as your action space is now the cross product of all of the other servos.

Alternatively you can use a "multi-headed" agent where each head chooses actions for a certain servo but there are pros and cons for both depending on your usecase.

If you have more advanced servos like Dynamixels which have high quality encoders, you'll have access to more advanced controls schemes. For instance, Dynamixels allow you to give actions in encoder space, angle space, and even velocity space. For example, you could give the action of "go to encoder value of 500" or "go to 90 degrees" or "move .5 radians/second". All of these approaches are useful for certain tasks. For humans controlling the arm using a joystick, the velocity based is the most intuitive and, in my experience, the same is true for RL agents using continuous control.

If you are using continuous control, you should normalize all of your action spaces within your agent then "unnormalize (?)" them before giving the actions to your servos. For instance if your servo velocity ranges from -3.5 rad/sec to +3.5 rad/sec, have your agent select actions in the range of [-1,1] then multiply by 3.5 to get the velocity.