Currently, we're trying to improve failure analysis capability when using neural nets. One thing we want to resolve is output variation between batched runs and non-batched runs.
For example, we wrote the following test code (hardware : NVIDIA T4 Inference card)
conv = torch.nn.Conv1d(40, 400, kernel_size=13, stride=1, padding=170, bias=True).cuda() with torch.cuda.amp.autocast(enabled=True): # This allows 16bit casting test_input = torch.rand(1, 40, 2536).cuda() test_output = conv(test_input) sample1 = test_output test_input = torch.cat([test_input, test_input], 0) test_output = conv(test_input) sample2 = test_output # test_output and test_output are equal diff = sample1 - sample2 diff = torch.abs(diff) diff = diff.sum() print(diff) # Expected output is zero.
But the output
diff is not zero. We also tried adding more jobs in batches and found that
- Results are different between batches (sample 1 and sample 2(outputs))
- In a given batch, results are all the same
I'd like to know what causes this issue with such a simple code, and if it's solvable or not while keeping high fp16 throughputs.