Alpha Go Zero contains several improvements compared to its predecessors. Architectural details of Alpha Go Zero can be seen in this cheat sheet.

One of those improvements is using a single neural network that calculates move probabilities and the state value at the same time, while the older versions used two separate neural networks. It has been shown that the merged neural network is more efficient according to the paper:

It uses one neural network rather than two. Earlier versions of AlphaGo used a “policy network” to select the next move to play and a ”value network” to predict the winner of the game from each position. These are combined in AlphaGo Zero, allowing it to be trained and evaluated more efficiently.

This seems counter intuitive to me, because from a software design perspective this violates the principle separation of concerns. That's why I am wondering, why this merge has been proven beneficial.

Can this technique - merging different tasks in a single neural network to improve efficiency - be applied to other neural networks in general or does this require certain conditions to work?


Why has this merge proven beneficial?

If you think about the shared Value/Policy network as consisting of a shared component (the Residual Network layers) with a Value and Policy component on top rather than Separation of Concerns it makes more sense.

The underlying premise is that the shared part of the network (the ResNet) provides a high-level generalization of the input (the game states leading up to the move) that is a good input representation for both the shallow Value and Policy networks.

When that is the case, we can reduce the computational load a lot by training a single shared ResNet and using it for two much simpler networks than training two ResNets for the Value and Policy. In their case, training the two together also improves regularisation and thus creates a more robust, general representation.

Specifically, the Alpha Go Zero paper by Silver et al., Mastering the Game of Go without Human Knowledge, states that:

Combining policy and value together into a single network slightly reduced the move prediction accuracy, but reduced the value error and boosted playing performance in AlphaGo by around another 600 Elo. This is partly due to improved computational efficiency, but more importantly the dual objective regularises the network to a common representation that supports multiple use cases.

Can this technique be applied in general or only in special cases?

Like common components in software libraries, it only makes sense when the problems you are trying to solve benefit from a shared representation.

You can use it if you are training classifiers for similar tasks, or training a new task with little data where you already have a classifier trained over a larger, similar dataset.

Outside of Go, it is often used in image recognition. Deep pre-trained networks such as the ones from the ImageNet ILSVRC competitions are commonly used as a starting point. They are classifiers that have been trained (for weeks!) on over a million images.

Then, say you want to create a network to recognize you favourite brand of bicycles, you start with the general image-recognition pipeline trained on ImageNet, chop of the last layers that do the actual classification ("it's a Border Collie") and add a small new classifier to pick out only the bicycles you care about.

Since the pre-trained classifier already provides high-level image concepts that are good building blocks for image recognition (it classifies 200 categories), this saves you a lot of training and makes for a very robust classifier.

Of course there are many cases where the problems do not have useful shared representations and thus no benefit from a combined network. Nevertheless, it is a useful tool in the right situations.

Look up Transfer Learning or Multi-Task Learning to learn more about this.

  • $\begingroup$ If one wants to use software engineering principles to analyze this neural network's architecture, I would also point out that the residual block network upon which the value and policy heads rest honor the DRY principle. By depending on the same component (the residual block network) to preprocess data before passing this data on to the other components in the pipeline (the value and policy heads), they guarantee that each will process the same representation of the original input. Duplicating this processing with two separate networks almost guarantees divergence over time. $\endgroup$ – sadakatsu Jan 24 at 17:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.