How many GPUs can these deep learning algorithms be parallelized across (batch parallelization)? [closed]

Question

The deep learning algorithms I would to know the limits of are:

1. CNTK
2. Caffe
3. TensorFlow
4. Torch7
5. Theano

For example: I've heard TensorFlow is near impossible to parallelize on 8 GPUs and above. So, in this case, the limit would be 8.

Answer 1

Disclaimer: this answer refers solely to TensorFlow, as my knowledge of the remaining frameworks is limited.

Where did you hear that TensorFlow is near impossible to parallelize on more than 8 GPUs? With a large enough network, any number of GPUs can be used to speed-up training, as shown in the tutorial. This snippet of code introduces the multiplication of 2 constants in 2 separate GPUs. This allows you to run the graph on both simultaneously.

for d in ['/gpu:2', '/gpu:3']:
    with tf.device(d):
        a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3])
        b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2])
        c.append(tf.matmul(a, b))

Even if there were some sort of limit on how many GPUs TensorFlow can use per process, TF can be distributed across processes and machines. Asynchronous algorithms (like A3C) can be spread across, for example, 16 machines (what I am doing in my laboratory), where each machine uses its resources (in my case, I use CPU, but the change to a GPU would be trivial).

With a complex enough network and a well implemented solution, you can take advantage of any hardware resources you have.

Answer 2

As an example for Caffee: Facebook parallelized the training of a Res-Net CNN over 256 GPUs