How can I train a neural network if I don't have enough data?

I have created a neural network that is able to recognize images with the numbers 1-5. The issue is that I have a database of 16x5 images which ,unfortunately, is not proving enough as the neural network fails in the test set. Are there ways to improve a neural network's performance without using more data? The ANN has approximately a 90% accuracy on the training sets and a 50% accuracy in the test ones.

Code:

clear
graphics_toolkit("gnuplot")
sigmoid = @(z) 1./(1 + exp(-z));
sig_der = @(y) sigmoid(y).*(1-sigmoid(y));

parse_image;   % This external f(x) loads the images so that they can be read.
%13x14
num=0;
for i=1:166
if mod(i-1,10)<=5 && mod(i-1,10) > 0
num=num+1;
end
end

function [cost, mid_layer, last_layer] = forward(w1,w2,data,sigmoid,i)
mid_layer(:,1)=sum(w1.*data(:,i));
mid_layer(:,2)=sigmoid(mid_layer(:,1));
last_layer(:,1)=sum(mid_layer(:,2).*w2);
last_layer(:,2)=sigmoid(last_layer(:,1));
exp_res=rem(i,5);
if exp_res==0
exp_res=5;
end
exp_result=zeros(5,1); exp_result(exp_res)=1;
cost = exp_result-last_layer(:,2);
end

function [w1, w2] = backprop(w1,w2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i)
delta(1:5) = cost;
delta(6:20) = sum(cost' .* w2,2);
w2 = w2 + 0.05 .* delta(1:5) .* mid_layer(:,2) .* sig_der(last_layer(:,1))';
w1 = w1 + 0.05 .* delta(6:20) .* sig_der(mid_layer(:,1))' .* data(:,i);
end

w1=rand(182,15)./2.*(rand(182,15).*-2+1);
w2=rand(15,5)./2.*(rand(15,5).*-2+1);

for j=1:10000
for i=[randperm(85)]
[cost, mid_layer, last_layer] = forward(w1,w2,data,sigmoid,i);
[w1, w2] = backprop(w1,w2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i);
cost_mem(j,i,:)=cost;
end
end


You can synthetically increase the number of samples. For example with augmentation or unsupervised adaption (Self-training). With augmentation you grant the system way more robustness so i would really recommend this. For example this github. The problem with such small database sizes is that your test-set is also very small and you cannot test properly if your network generalizes well, or just overfits.

You can try transfer learning with another larger network to adapt those feature extractors and use them on ur problem. That may work better than training a new one from scratch with so less labeled images. Hope i could help at least a little, stay tuned.

• Thanks. I was wondering if there was a small tweak you could make to the back propagation algorithm to improve training. Apr 3, 2020 at 15:48
• Please if you find that instant contact me haha :D. Mhm maybe consider dropout, or adding noise on your system. Meaning add gaussian noise to the weigths itself, works quite good in self-learning and GAN´s, also stochastic depth is quite nice for the same kind of tweaking Apr 3, 2020 at 15:50
• Ill try the gaussian noise. Thanks! Apr 3, 2020 at 15:51
• i like the easy keras implementation. Maybe you want to try it here: keras.io/layers/noise Apr 3, 2020 at 15:52
• I'm not a fan of python. I usually don't like using libraries either as they aren't very instructive but thanks a lot . I'll still check it out. Apr 3, 2020 at 15:53

In theory, yes, using synthetic data generation. This involves applying transformations to the original images to generate new 'unique' images. Some standard techniques include rotating, flipping, stretching, zooming or brightening. Obviously not all of these make sense depending on the data. In your problem, zooming, stretching and brightening could be used but flipping should not. Rotation could work but only for small angles.

Generally this is implemented by replacing the dataset for each epoch of training. Therefore, the number of images used in each training iteration is the same but the images themselves have been altered.

In practice, it's not a magic bullet. The reason a larger dataset generally yields better models is because the probability of a new feature falling within the feature distribution of the training data is higher. With synthetic data generation the new features are only marginally different to the original so even if the number of images to train on is increased, the feature distributions are not that different. There is a lot of variation in handwritten numbers so it would be very hard to guess how effective this would be without trying it.