The topology of a neural network can have a significant impact on the performance of
The most important factor is the
number of layers and the
connectivity between them.
Simple topologies require less data movement and can be more easily parallelized. a shallower network with fewer layers will often be faster to train on a
TPU than a deeper network with more layers. This is because each layer in a neural network must be fully connected to the previous and next layers, and a deep network will have many more connections than a shallow network.
Additionally, the activation function used for each layer can also impact performance.
ReLU is a common activation function that is often used in networks that are accelerated by
GPU-based acceleration of CNN
GPU is widely used in neural network applications due to a large number of ALU units which helps in faster data processing (multiplication and summation operations in
NN), and also the
GPU caches, which help in data reuse.
GPU is capable of merging multiple data access requests using the controllers, and it helps in massive parallel and pipelined processing.
GPU is a temporal architecture paradigm with a large number of
ALUs, but the ALUs lack direct data communication, and they communicate using direct memory access.
GPU has around $3,000–5,000$
ALU But the Von Neumann bottleneck exists in
GPU due to the access to registers and the shared memory for intermediate data storage in every
GPU has specialized libraries for
CNN acceleration like fbfft (Vasilache et al., 2015). While using a high working set, the shared memory cannot be used, and there is a need for global memory access in
GPU, and this leads to more memory footprints and memory access.
TPU-based acceleration of CNN
TPU is a custom-made
ASIC with a matrix processor which is specially designed for neural networks. it effectively handles the addition and multiplication in neural nets at a very high speed with very little power consumption.
The von Neumann bottleneck in
GPU is overcome in
TPU with the systolic array structure. TPU v2 single processor has
128 × 128 systolic arrays with
Systolic array in TPU helps in data reuse which makes the performance high and execution energy efficient in
The two-dimensional multiply unit helps in matrix multiplication faster compared to the one-dimensional multiply units in
TPU, eight-bit integers are used in place of the
32-bit floating-point operations, and this makes the computations faster and memory efficient.
TPU drops features that are not used in the neural network, which helps in saving energy.
CNN implementation in
TPU will have both
CPU usage in parallel to run the linear and non-linear elements in
CNN the convolution and classification layer is executed in
TPUsince it is a GEMM operation, and the Pooling and Flattening are executed in the
Here is the paper that analyzed CNN model performance for the three-image processing application in GPU/TPU platforms in Colab for various batch sizes. The analysis was done by
varying the final feed-forward network and the hidden layers, and this gives an inference on how the
performance is affected when
the model structure changes
Fig(a): Training time for single and multiple convolutional layer networks.
In the paper, analysis was done using a single convolutional layer followed by all other layers for the mask detection application for binary class, and the training time was analyzed for GPU and TPU for both networks. The training time increases when the convolutional layers are removed because the number of nodes gets more, and thereby training time increases. The training time for Single layer convolution and multiple-layer Convolution for different batch sizes is shown in
The analysis clearly shows that the time decreases when the number of convolutions increases due to a reduction in the number of nodes. The training time is less for the multiple layers CNN compared to single-layer CNN and also with an increase in the batch size.
Fig(b): Execution time for multi and binary classes.
The overall training time for each of the three applications was in GPU and TPU for both binary, and multiple classifications were analyzed and shown in
Fig(b), it is clear that compared to
GPU has a low time for execution of the CNN. This occurs due to the bottleneck that occurs in TPU due to the in-between CPU access.
GPU: GPU performs well for small batches and gives better flexibility and easy programming. For small data, batch sizes GPU fits better due to the execution pattern in wraps and scheduling id easy on-stream multiprocessors. For large dataset and network models, GPU performs well by optimizing memory reuse. In fully connected neural networks, weight reuse is less, so as the model size increases, this leads to high memory traffic. In GPU, the memory bandwidth makes it practical for applications with memory requirements. Large neural networks work better on GPU compared to CPU due to the extra parallelism feature. For fully connected neural networks, GPU works better compared to CPU, but for large batch sizes, TPU performs well.
TPU: TPU performs well on CNN with large batches to give high throughput in training time using the systolic array structure. Large batches of data are needed for the full utilization of the matrix multiply units in the systolic array of TPU. In CNN, the speedup increases with batch size. For enormous batch sizes and complex CNN, TPU is the best because of the spatial reuse characteristics of CNNs. But in fully connected networks, the weight reuse is less, and so TPU is not preferred.
Different neural network topologies may require different amounts of resources (memory, computational power, etc.), which could affect the efficiency of hardware acceleration. There is no easy rule of thumb, however, as it depends on the specific architecture and implementation.