Tag Info

4

There are lots of ways that RL agents can fail to learn properly, so you are faced with a little bit of experimentation and maybe bug hunting unfortunately. However, from the description you have given in the question and comments, I can make a few observations and guesses about where to look: Your metric of average reward against a random player is ...

3

The AlphaZero paper mentions an "evaluation" step that seems to deal with the the problem similar to yours: ... we evaluate each new neural network checkpoint against the current best network $f_{\theta_*}$ before using it for data generation ... Each evaluation consists of 400 games ... If the new player wins by a margin of > 55% (to avoid ...

3

If you are running self-play in a two player zero sum game, then you can do the following: Arbitrarily decide the reward scheme for winning, drawing, losing is +1, 0, -1 for Player A. Have Player A's goal to maximise reward, and Player B's goal to minimise reward. This means you can combine both players' view of the values of positions and plays into a ...

3

When one agent makes a move, that move should be perceived as part of the "state transition" executed "by the environment" from the perspective of the other agent. For example, suppose that, as a "neutral third party" we view the game as follows, as a sequence of states, actions and a terminal reward. I use A to denote actions selected by the first player, ...

2

When in an environment with competing agents, from the perspective of each agent, the environment becomes non-markovian. That occurs because each agent is constantly adapting its own strategy to other's actions, so a transition that occurred to a pair (s,a) before, resulting in a positive reward, might result in zero or negative reward in future iterations ...

1

Generally, "perfect information" is not a formal trait of MDPs. There is a concept of the Markov property, but it only loosely coincides with "perfect information". For instance it is OK for there to be unknown/hidden state, provided it behaves effectively randomly (when revealed, it is drawn from a consistent distribution). An example ...

1

Both players are represented by the exact same network with the exact same weights(similar to AplhaGO, AlphaGoZero and AlphaZero). So, they will both behave identical. Because you only have a single network, MuZero can not learn two different policies, but only one. You can also think of this in the following way: MuZero actually learn to play only with ...

1

To solve this, I now train batches of models (~ 10 models per batch), which are then used in group as a new opponent, This seems quite a reasonable approach on the surface, but possibly the agents will still lose generalisation if the solutions in each generation are too similar. It also looks like from your experiment that learning progress is too slow. ...

1

However, I’m not sure which policy is saved The policy from the Monte Carlo tree search is stored, as we can get the policy estimate from the network later by passing the given state through the network, which is used to calculate the cross entropy loss to update the network's policy (summed with Mean squared error loss between value head's prediction and ...

1

Chapter 1 of Sutton & Barto, doesn't introduce the full version Q learning, and you are probably not expected to explain the full distribution of values at that stage. Probably what you are expected to notice is that the maximum Q values out of possible next states - after training/convergence - should represent the agent's best choice of move. What the ...

1

Keras/Tensorflow are mostly used of developing/training/deploying neural networks. For descision making problems, if you want to use machine learning, reinforcement learning is in most cases applied. Some reinforcement learning methods use neural networks (and therfore tensorflow) internally. You can check baseline implementations of different methods here. ...

Only top voted, non community-wiki answers of a minimum length are eligible