-1
$\begingroup$

It seems no difference for cpu but not for gpu

enter image description here enter image description here

$\endgroup$
2
  • 1
    $\begingroup$ The GPU times are tiny, literally 1000s of times faster than CPU ones, so are going to be heavily influenced by random fluctuations and any side issues the Python interpreter has. I would suggest running within a proper benchmarking function which will run them multiple times to get a decent average. $\endgroup$ Commented Nov 27 at 10:56
  • $\begingroup$ I don't understand your question, ...float numbers rather than zeros?, what do you mean by that? zeros can be represented in float or integer, but not both. $\endgroup$ Commented Nov 28 at 12:28

2 Answers 2

1
$\begingroup$

Due to their architecture, GPUS is faster on floating numbers rather than integers and binary numbers like one and zero. they are designed to perform floating point arithmetic operations such as matrix operations, image rending, graphical processing, and scientific operations.

and the GPUs use the SIMD (Single Instruction Multiple Data) model, where a single operation runs on multiple data points. For that type of task, floating points are mostly suitable.

The pipeline in a GPU is highly optimized for floating-point operations, including addition, multiplication, and division. Processing integers or zeros might not fully utilize these pipelines, resulting in under-utilization of GPU resources.

While zeros are computationally less expensive in some scenarios (e.g., sparse matrices), GPUs often don't have specific optimizations for them unless explicitly handled in software. Floating-point operations are given more attention for general workloads

In simple words, GPUs mostly run fast on floating numbers rather than zeros because they are designed to work on floating operations :), and working with zero will not fully utilize the gpu usage.

$\endgroup$
3
  • $\begingroup$ While I agree with you in general, torch.zeros returns the same data type as torch.randn. The default type is float32. $\endgroup$
    – NikoNyrh
    Commented Nov 30 at 2:25
  • $\begingroup$ Actually torch doesn't even support multiplying int32 and float32 matrices together! expected mat1 and mat2 to have the same dtype, but got: int != float. And at least on my setup multiplying int32 matrices together isn't implemented either: "addmm_cuda" not implemented for 'Int'. $\endgroup$
    – NikoNyrh
    Commented Nov 30 at 11:19
  • $\begingroup$ Yes @NikoNyrh, you are correct about the torch. zeros and torch. since both are using the float, I think the issue is due to not using torch.cuda.synchronize() $\endgroup$ Commented Dec 2 at 4:15
1
$\begingroup$

torch.zeros returns an array of torch.float32, just as torch.randn, so I don't think Keval's explanation is correct.

Here is a simple benchmarking script:

import time
import torch
import numpy as np

device = 'cuda'
a = torch.zeros((10000,10000)).to(device)
b = torch.randn((10000,10000)).to(device)
c = torch.randn((10000,10000)).to(device)

times = [time.time()]
while times[-1] - times[0] < 3:  # Run for 3 seconds
    tmp1 = a @ b
    times.append(time.time())
    torch.cuda.synchronize()
    times.append(time.time())

    tmp2 = c @ b
    times.append(time.time())
    torch.cuda.synchronize()
    times.append(time.time())



times = np.array(times)
times = (times[1:] - times[:-1]).reshape((len(times) // 4, 4))


np.log10(times).round(2)[:10]
'''
array([[-2.97, -1.27, -3.78, -1.04],
       [-3.85, -1.27, -3.89, -1.25],
       [-4.  , -1.22, -3.92, -1.16],
       [-3.87, -1.09, -3.73, -1.04],
       [-3.97, -0.97, -3.75, -0.98],
       [-3.69, -1.  , -4.  , -0.97],
       [-3.85, -1.02, -3.95, -0.97],
       [-3.88, -0.99, -3.93, -0.96],
       [-3.85, -0.98, -3.69, -0.98],
       [-3.95, -0.99, -3.95, -0.97]])
'''

Note how calling a @ b or c @ b takes barely any time, the torch API is asynchronous (we can run GPU and Python code in parallel). Thus we need to call torch.cuda.synchronize() to measure how long the calculation took.

I actually made a big mistake on the first version of the script, omitting the synchronization call.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .