# Non-determinism with mixed precision?

Currently, we're trying to improve failure analysis capability when using neural nets. One thing we want to resolve is output variation between batched runs and non-batched runs.

For example, we wrote the following test code (hardware : NVIDIA T4 Inference card)

conv = torch.nn.Conv1d(40, 400, kernel_size=13, stride=1, padding=170, bias=True).cuda()
with torch.cuda.amp.autocast(enabled=True): # This allows 16bit casting
test_input = torch.rand(1, 40, 2536).cuda()
test_output = conv(test_input)
sample1 = test_output[0]

test_input = torch.cat([test_input, test_input], 0)
test_output = conv(test_input)
sample2 = test_output[0]    # test_output[0] and test_output[1] are equal

diff = sample1 - sample2
diff = torch.abs(diff)
diff = diff.sum()
print(diff) # Expected output is zero.


But the output diff is not zero. We also tried adding more jobs in batches and found that

• Results are different between batches (sample 1 and sample 2(outputs))
• In a given batch, results are all the same

I'd like to know what causes this issue with such a simple code, and if it's solvable or not while keeping high fp16 throughputs.