1
$\begingroup$

Currently, we're trying to improve failure analysis capability when using neural nets. One thing we want to resolve is output variation between batched runs and non-batched runs.

For example, we wrote the following test code (hardware : NVIDIA T4 Inference card)

conv = torch.nn.Conv1d(40, 400, kernel_size=13, stride=1, padding=170, bias=True).cuda()
with torch.cuda.amp.autocast(enabled=True): # This allows 16bit casting
   test_input = torch.rand(1, 40, 2536).cuda()
   test_output = conv(test_input)
   sample1 = test_output[0]

   test_input = torch.cat([test_input, test_input], 0)
   test_output = conv(test_input)
   sample2 = test_output[0]    # test_output[0] and test_output[1] are equal

   diff = sample1 - sample2
   diff = torch.abs(diff)
   diff = diff.sum()
   print(diff) # Expected output is zero.

But the output diff is not zero. We also tried adding more jobs in batches and found that

  • Results are different between batches (sample 1 and sample 2(outputs))
  • In a given batch, results are all the same

I'd like to know what causes this issue with such a simple code, and if it's solvable or not while keeping high fp16 throughputs.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.