3
$\begingroup$

There are reinforcement learning papers (e.g. Metacontrol for Adaptive Imagination-Based Optimization) that use (apparently, interchangeably) the term control or action to refer to the effect of the agent on the environment at each time step.

Is there any difference between the terms control or action or are they (always) used interchangeably? If there is a difference, when is one term used as opposed to the other?

The term control likely comes from the field of optimal control theory, which is related to reinforcement learning.

$\endgroup$
  • $\begingroup$ Another question to add to our archive! I am looking for answers that ideally support the given statements (maybe by linking to external resources). I could also conclude that the terms are used interchangeably, given my experience, so that's why I am looking for answers that support the provided statements. $\endgroup$ – nbro Jun 28 at 18:02
2
$\begingroup$

There's no difference. As they too often do, ML researchers take concepts from other disciplines, conveniently forget to cite sources and change the terminology, leading to much confusion. RL is a textbook example (pun intended). Optimal control researchers have been studying very similar problems long before RL ones, and used standard symbols and terms ($x$ for states, $u$ for controls). Then RL researchers came and changed just about everything.

See the paper A Tour of Reinforcement Learning: The View from Continuous Control (2018), by Benjamin Recht, which discusses reinforcement learning from a control and optimization perspective.

See also this tweet https://twitter.com/beenwrekt/status/1134536093980864514?s=21 (by Benjamin Recht) regarding the presentation of Sham Kakade.

$\endgroup$
-2
$\begingroup$

There is a huge difference; it has to do with how to understand a control problem. The traditional form of optimal control is located within the number crunching domain. A control statement is used to influence a system with a mathematical value. For example we can feed the number “-1” into the plant and get a certain result as a consequence. The result depends on the nonlinear differential equation which describes the inner working of the system.

In contrast, the term action is used in more recent publications which are trying to abstract from the mathematical control theory in favor of a linguistic description of a problem. Action based reinforcement learning is strongly connected to natural language parsing. An action primitive isn't a mathematical value like “-1” but it's a grounded statement like “move left”. The hope is, that linguistic grounded actions have a better modeling performance for complicated systems.

In most cases, actions aren't feed into a mathematical equation but into a grammar driven model. An action like “moveleft” has much to do with language parsing, but only little with a classical pid control problem. Actions are used for solving complex problems, while control signals are useful for non- compound tasks.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.