I'm starting to learn more about mixed-precision training, and I'm in particular confused about point-wise operations. In the original article (link), the authors mention, citing:
Point-wise operations, such as non-linearities and element-wise matrix products, are memory-bandwidth limited. Since arithmetic precision does not impact the speed of these operations, either FP16 or FP32 math can be used.
What I'm wondering is - don't element-wise matrix products, for example, take more time to compute since FP32 product is computationally heavier than FP16 product? And even if the FP16 product takes the same amount of time as the FP32 product, we're still limited by the same memory bandwidth, meaning that we'll be able to perform twice as many operations concurrently?