Positive bias causes the calculation of incorrect gradients

Question

I have a data set with a positive bias (an image, where the values range from 0 to 1), that seems to be causing my network to calculate incorrect gradients.

If I just use the raw image as input, of shape (1,28,28), gradient checking fails on the convolution layers, producing the following:

conv0: 1.1387914962261193e-06 1.4299464517486544     77.73286194827031
conv1: 7.635784666538102e-06  0.13917264577152977    2.5142484667483797e-06
conv2: 5.935327486981425e-07  2.706137838441397      1.6290818256837689e-06
fc0:   6.053556079013436e-07  1.1083181748299355e-05 2.0993768208698443e-06
fc1:   2.5819794868905293e-08 4.499689509732119e-06  1.108312270578618e-05

Where the first value is the absolute difference between the calculated weight updates, and the gradient tested weight updates, second being bias', and third being the calculated gradient to be passed to the previous layer. If this value is less than 0.1, it's a good indication the gradients are correct, as I use an epsilon of 1e-5.

Strangely if I use a randomly initialised input: inpt = np.random.randn(*im.shape)) using the same seed as above for network initialisation, then I get the following differences:

conv0: 0.002403805867625952   1.1140792221336904e-05 0.0004023637406479723
conv1: 0.0011835450640199086  6.134154765691235e-06  0.00014655404401104377
conv2: 0.00021913017743990792 0.0032477812666108496  7.169014825140573e-05
fc0:   0.04510183914723059    6.425385741260989e-06  1.709243452626911e-05
fc1:   0.0014606421065230097  1.9235668613026775e-06 5.995613024721222e-06

Would anyone have any idea what be causing this? I'm truly at a loss here, as I do the same procedure regardless of the numbers in the input, using the following to find the gradients to be passed back to the previous layer (for conv layers):

t = np.pad(grad, ((0,0), (self.xpad[1],self.xpad[0]), (self.ypad[1],self.ypad[0])))
self.gradient = np.zeros(self.incoming.output.shape)
for j in range(self.gradient.shape[0]):
    for i in range(t.shape[0]):
        self.gradient[j] += signal.convolve(t[i], self.kernels[i,j], mode='valid')

(This would be signal.correlate if I was using a convolution on the forward pass, but I'm not. Important to note that a convolutional neural network actually uses correlation on the forward pass, and only convolves on the backward)

EDIT:

I have done some more testing and found that if I change epsilon even slightly, I get MASSIVELY different results for the tested gradient. It seems decreasing epsilon increases the magnitude of the gradients exponentially. This is very strange behaviour, and I'm really not sure what is causing it. Could it be a discontinuity in ReLU where somewhere along the way I happen upon exactly 0?

Answer 1

Ok, so I was right in assuming it was the discontinuity at x=0 for relu(x). The positive bias actually had nothing to do with the incorrect gradients. As for why in my edit I was getting such massive differences between gradient values, it was because I initialised the input to be integers, and upon adding a decimal, no matter how small (> 0 though), numpy would jump entire integers, so -2 + 0.0001 = -1.

This is reproduced by:

>>> a = np.random.randint(-3,3,(1,3,3))
>>> a
array([[[-2,  1, -2],
        [-2, -1, -3],
        [-3,  0,  1]]])
>>> a[0][0][0] += 0.0001
>>> a
array([[[-1,  1, -2],
        [-2, -1, -3],
        [-3,  0,  1]]])

The problem with my input is large amounts of it were 0, so it produced a bunch of discontinuous gradients. As such, I simply check both sides of the graph (have epsilon positive then negative), and if they disagree, I set the value to 0, as in that case I'd just ignore any gradient passing through that node.