0
$\begingroup$

If you have a game and you are training an AI there seems to be two ways to do it.

First you take the game-state and a possible move and evaluate whether this move would be good or bad:

(1) GAME_STATE + POSSIBLE_MOVE --> Good or bad?

The second is to take the game state and get probabilities of every conceivable move:

(2) GAME_STATE ---> Probabilities for each move

It seems that both models are used. I.e. in language and RNN might use (2) to find the probabilities for each next word or letter. But AlphaZero might use (1). Noting also that in a game like chess GAME_STATE + POSSIBLE_MOVE = NEW_GAME_STATE. Whereas in some games you might not know the result of your move.

Which do you think is the best method? Which is the best way to do AI? Or some combination of the two?

$\endgroup$
  • 1
    $\begingroup$ Actually AlphaGo uses 1 + 2 (plus a look-ahead search to help refine them), and this is relatively common too (search "Actor-Critic"). $\endgroup$ – Neil Slater Apr 15 '18 at 19:59
  • $\begingroup$ Ah, I see! I don't think AlphaZero uses Actor-Critic though? $\endgroup$ – zooby Apr 15 '18 at 21:57
  • $\begingroup$ Yes you are right, it is not Actor-Critic (as far as I understand it). It does however both generate policies directly and evaluate board positions. The tree search (MCTS) is what links the two. $\endgroup$ – Neil Slater Apr 16 '18 at 7:03
1
$\begingroup$

Which do you think is the best method?

As with most machine learning, each approach has its strengths and weaknesses, and other than a little bit of intuition:

  • Policy-based methods are strong in large or continuous action spaces, and/or where there is a simple relationship between state and optimal action. E.g. controlling a robotic arm with continuous action space.

  • Value-based methods are strong where there is a simple relationship between state and value under an optimal policy. E.g. in a maze game.

It may not always be clear which is best, in which case experimentation is required. If using neural networks, there will then be a large number of hyper-parameters on each approach, so it may be hard to come to a strict conclusion about which is better. Although you can include "easy to find a working model" or "robust for a range of hyper-parameter values" as benefits of any type of model if you wish - these are important practical benefits of any approach, developers rarely want to do 100s of experiments to tune a learning rate parameter for example.

Which is the best way to do AI? Or some combination of the two?

Actor-Critic, as seen in Asynchronous Advantage Actor Critic. A3C and A2C (deterministic variant of A3C) is producing current state-of-the-art results in video games. This is a combination of both approaches, where the agent maintains two related models - one, the Actor, generates a policy directly by looking at the state, and the second, the Critic, tracks the estimated value of each state. Often, these two models share some parameters - e.g. using neural networks, the initial layers may be the same for both.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.