Currently, we're trying to improve failure analysis capability when using neural nets. One thing we want to resolve is output variation between batched runs and non-batched runs.

For example, we wrote the following test code (hardware : NVIDIA T4 Inference card)

conv = torch.nn.Conv1d(40, 400, kernel_size=13, stride=1, padding=170, bias=True).cuda()
with torch.cuda.amp.autocast(enabled=True): # This allows 16bit casting
   test_input = torch.rand(1, 40, 2536).cuda()
   test_output = conv(test_input)
   sample1 = test_output[0]

   test_input = torch.cat([test_input, test_input], 0)
   test_output = conv(test_input)
   sample2 = test_output[0]    # test_output[0] and test_output[1] are equal

   diff = sample1 - sample2
   diff = torch.abs(diff)
   diff = diff.sum()
   print(diff) # Expected output is zero.

But the output diff is not zero. We also tried adding more jobs in batches and found that

  • Results are different between batches (sample 1 and sample 2(outputs))
  • In a given batch, results are all the same

I'd like to know what causes this issue with such a simple code, and if it's solvable or not while keeping high fp16 throughputs.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.