0
$\begingroup$

Before a robot can act with meaning, some planning is needed. The idea is, that the decision making process is independent from action. The task of figuring out what the best decision is, was introduced under the term Cybernetics and is focused on intelligence in animals and artificial systems. In a robot, the act of thinking is realized with computer programs.

If we are researching how exactly a computer program determines what the next action for a robot is, the term reinforcement learning is often referenced. The idea is, to store all situations in a q-table and if the robot is in a certain situation he can look into the q-matrix and sees in the row what action is the right one. This is called a policy because it answers the question what action is needed in a certain state.

Some researchers have recognized that Q-learning is not the best way for dealing with complex problems. A policy which is similar to a lookup table will provide only simple formed answers to difficult challenges. The more elaborated form of storing knowledge is called an indirect policy. The idea is here not to store state-action values in a table but use a prediction model. The first thing what the robot software is answering are potential actions and for each action an outcome is calculated. The predicted result of the actions is stored in graph and the best node is identified with a solver.

In a concrete situation a direct policy knows what the correct action is, but the indirect policy don't. The indirect policy has to figure out first what the prediction of future states will be. My question is: is an indirect policy which is using a prediction model superior to a direct policy known as a q-table?

$\endgroup$
  • $\begingroup$ "The idea is, to store all situations in a q-table and if the robot is in a certain situation he can look into the q-matrix and sees in the row what action is the right one. This is called a policy because it answers the question what action is needed in a certain state.", that is not really called policy. Looking at the Q-table and seeing which one is the "right action" is not the definition of a policy. A policy can by any function which lets you behave (or take actions) in an environment. Nonetheless, a policy can be derived from the Q-table. $\endgroup$ – nbro Feb 12 at 11:41
  • $\begingroup$ "Some researchers have recognized that Q-learning is not the best way for dealing with complex problems.", which researchers are you talking about? Q-learning has actually be used to solve complex problems (like playing Atari games). $\endgroup$ – nbro Feb 12 at 11:42
  • 1
    $\begingroup$ "The more elaborated form of storing knowledge is called an indirect policy". I've never heard of the term "indirect policy". Where did you find it? $\endgroup$ – nbro Feb 12 at 11:44
  • $\begingroup$ "The first thing what the robot software is answering are potential actions and for each action an outcome is calculated." What outcome are you talking about? What should this outcome represent? $\endgroup$ – nbro Feb 12 at 11:45
  • $\begingroup$ "The predicted result of the actions is stored in graph and the best node is identified with a solver", I've never heard of this. Where did you learn about this method? Can you please cite a source? $\endgroup$ – nbro Feb 12 at 11:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.