The topology of a neural network can have a significant impact on the performance of GPU and TPU acceleration.
The most important factor is the number of layers and the connectivity between them.
Simple topologies require less data movement and can be more easily parallelized. a shallower network with fewer layers will often be faster to train on a GPU or TPU than a deeper network with more layers. This is because each layer in a neural network must be fully connected to the previous and next layers, and a deep network will have many more connections than a shallow network.
Additionally, the activation function used for each layer can also impact performance. ReLU is a common activation function that is often used in networks that are accelerated by GPUs or TPUs.
GPU-based acceleration of CNN
GPU is widely used in neural network applications due to a large number of ALU units which helps in faster data processing (multiplication and summation operations in NN), and also the GPU caches, which help in data reuse.
The GPU is capable of merging multiple data access requests using the controllers, and it helps in massive parallel and pipelined processing.
GPU is a temporal architecture paradigm with a large number of ALUs, but the ALUs lack direct data communication, and they communicate using direct memory access.
GPU has around $3,000–5,000$ ALU But the Von Neumann bottleneck exists in GPU due to the access to registers and the shared memory for intermediate data storage in every ALU operation.
The GPU has specialized libraries for CNN acceleration like fbfft (Vasilache et al., 2015). While using a high working set, the shared memory cannot be used, and there is a need for global memory access in GPU, and this leads to more memory footprints and memory access.
TPU-based acceleration of CNN
TPU is a custom-made ASIC with a matrix processor which is specially designed for neural networks. it effectively handles the addition and multiplication in neural nets at a very high speed with very little power consumption.
The von Neumann bottleneck in CPU and GPU is overcome in TPU with the systolic array structure. TPU v2 single processor has 16-bit two 128 × 128 systolic arrays with 32,768 ALUs.
Systolic array in TPU helps in data reuse which makes the performance high and execution energy efficient in CNN.
The two-dimensional multiply unit helps in matrix multiplication faster compared to the one-dimensional multiply units in CPU and GPU.
In TPU, eight-bit integers are used in place of the 32-bit floating-point operations, and this makes the computations faster and memory efficient.
Unlike CPU and GPU TPU drops features that are not used in the neural network, which helps in saving energy.
CNN implementation in TPU will have both TPU and CPU usage in parallel to run the linear and non-linear elements in CNN.
In CNN the convolution and classification layer is executed in TPUsince it is a GEMM operation, and the Pooling and Flattening are executed in the CPU.
Here is the paper that analyzed CNN model performance for the three-image processing application in GPU/TPU platforms in Colab for various batch sizes. The analysis was done by varying the final feed-forward network and the hidden layers, and this gives an inference on how the performance is affected when the model structure changes
Fig(a): Training time for single and multiple convolutional layer networks.
In the paper, analysis was done using a single convolutional layer followed by all other layers for the mask detection application for binary class, and the training time was analyzed for GPU and TPU for both networks. The training time increases when the convolutional layers are removed because the number of nodes gets more, and thereby training time increases. The training time for Single layer convolution and multiple-layer Convolution for different batch sizes is shown in Fig(a).
The analysis clearly shows that the time decreases when the number of convolutions increases due to a reduction in the number of nodes. The training time is less for the multiple layers CNN compared to single-layer CNN and also with an increase in the batch size.
Fig(b): Execution time for multi and binary classes.
The overall training time for each of the three applications was in GPU and TPU for both binary, and multiple classifications were analyzed and shown in Fig(b).
From Fig(b), it is clear that compared to TPU, GPU has a low time for execution of the CNN. This occurs due to the bottleneck that occurs in TPU due to the in-between CPU access.
GPU: GPU performs well for small batches and gives better flexibility and easy programming. For small data, batch sizes GPU fits better due to the execution pattern in wraps and scheduling id easy on-stream multiprocessors. For large dataset and network models, GPU performs well by optimizing memory reuse. In fully connected neural networks, weight reuse is less, so as the model size increases, this leads to high memory traffic. In GPU, the memory bandwidth makes it practical for applications with memory requirements. Large neural networks work better on GPU compared to CPU due to the extra parallelism feature. For fully connected neural networks, GPU works better compared to CPU, but for large batch sizes, TPU performs well.
TPU: TPU performs well on CNN with large batches to give high throughput in training time using the systolic array structure. Large batches of data are needed for the full utilization of the matrix multiply units in the systolic array of TPU. In CNN, the speedup increases with batch size. For enormous batch sizes and complex CNN, TPU is the best because of the spatial reuse characteristics of CNNs. But in fully connected networks, the weight reuse is less, and so TPU is not preferred.
Different neural network topologies may require different amounts of resources (memory, computational power, etc.), which could affect the efficiency of hardware acceleration. There is no easy rule of thumb, however, as it depends on the specific architecture and implementation.
