I have a data set with a positive bias (an image, where the values range from 0 to 1), that seems to be causing my network to calculate incorrect gradients.
If I just use the raw image as input, of shape
(1,28,28), gradient checking fails on the convolution layers, producing the following:
conv0: 1.1387914962261193e-06 1.4299464517486544 77.73286194827031 conv1: 7.635784666538102e-06 0.13917264577152977 2.5142484667483797e-06 conv2: 5.935327486981425e-07 2.706137838441397 1.6290818256837689e-06 fc0: 6.053556079013436e-07 1.1083181748299355e-05 2.0993768208698443e-06 fc1: 2.5819794868905293e-08 4.499689509732119e-06 1.108312270578618e-05
Where the first value is the absolute difference between the calculated weight updates, and the gradient tested weight updates, second being bias', and third being the calculated gradient to be passed to the previous layer. If this value is less than 0.1, it's a good indication the gradients are correct, as I use an epsilon of
Strangely if I use a randomly initialised input:
inpt = np.random.randn(*im.shape)) using the same seed as above for network initialisation, then I get the following differences:
conv0: 0.002403805867625952 1.1140792221336904e-05 0.0004023637406479723 conv1: 0.0011835450640199086 6.134154765691235e-06 0.00014655404401104377 conv2: 0.00021913017743990792 0.0032477812666108496 7.169014825140573e-05 fc0: 0.04510183914723059 6.425385741260989e-06 1.709243452626911e-05 fc1: 0.0014606421065230097 1.9235668613026775e-06 5.995613024721222e-06
Would anyone have any idea what be causing this? I'm truly at a loss here, as I do the same procedure regardless of the numbers in the input, using the following to find the gradients to be passed back to the previous layer (for
t = np.pad(grad, ((0,0), (self.xpad,self.xpad), (self.ypad,self.ypad))) self.gradient = np.zeros(self.incoming.output.shape) for j in range(self.gradient.shape): for i in range(t.shape): self.gradient[j] += signal.convolve(t[i], self.kernels[i,j], mode='valid')
(This would be
signal.correlate if I was using a convolution on the forward pass, but I'm not. Important to note that a convolutional neural network actually uses correlation on the forward pass, and only convolves on the backward)
I have done some more testing and found that if I change epsilon even slightly, I get MASSIVELY different results for the tested gradient. It seems decreasing epsilon increases the magnitude of the gradients exponentially. This is very strange behaviour, and I'm really not sure what is causing it. Could it be a discontinuity in ReLU where somewhere along the way I happen upon exactly 0?