How can I train a neural network if I don't have enough data?

Question

I have created a neural network that is able to recognize images with the numbers 1-5. The issue is that I have a database of 16x5 images which ,unfortunately, is not proving enough as the neural network fails in the test set. Are there ways to improve a neural network's performance without using more data? The ANN has approximately a 90% accuracy on the training sets and a 50% accuracy in the test ones.

Code:

clear
graphics_toolkit("gnuplot")
sigmoid = @(z) 1./(1 + exp(-z));
sig_der = @(y) sigmoid(y).*(1-sigmoid(y));


parse_image;   % This external f(x) loads the images so that they can be read. 
%13x14
num=0;
for i=1:166
  if mod(i-1,10)<=5 && mod(i-1,10) > 0
    num=num+1;
    data(:,num) = dlmread(strcat("/tmp/",num2str(i)))(:);
  end
end



function [cost, mid_layer, last_layer] = forward(w1,w2,data,sigmoid,i)
  mid_layer(:,1)=sum(w1.*data(:,i));
  mid_layer(:,2)=sigmoid(mid_layer(:,1));
  last_layer(:,1)=sum(mid_layer(:,2).*w2);
  last_layer(:,2)=sigmoid(last_layer(:,1));
  exp_res=rem(i,5);
  if exp_res==0
    exp_res=5;
  end
  exp_result=zeros(5,1); exp_result(exp_res)=1;
  cost = exp_result-last_layer(:,2);
end

function [w1, w2] = backprop(w1,w2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i)
  delta(1:5) = cost;
  delta(6:20) = sum(cost' .* w2,2);
  w2 = w2 + 0.05 .* delta(1:5) .* mid_layer(:,2) .* sig_der(last_layer(:,1))';
  w1 = w1 + 0.05 .* delta(6:20) .* sig_der(mid_layer(:,1))' .* data(:,i);
end

w1=rand(182,15)./2.*(rand(182,15).*-2+1);
w2=rand(15,5)./2.*(rand(15,5).*-2+1);

for j=1:10000
  for i=[randperm(85)]
    [cost, mid_layer, last_layer] = forward(w1,w2,data,sigmoid,i);
    [w1, w2] = backprop(w1,w2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i);
    cost_mem(j,i,:)=cost;
  end
end

Answer 1

You can synthetically increase the number of samples. For example with augmentation or unsupervised adaption (Self-training). With augmentation you grant the system way more robustness so i would really recommend this. For example this github. The problem with such small database sizes is that your test-set is also very small and you cannot test properly if your network generalizes well, or just overfits.

You can try transfer learning with another larger network to adapt those feature extractors and use them on ur problem. That may work better than training a new one from scratch with so less labeled images. Hope i could help at least a little, stay tuned.

Answer 2

In theory, yes, using synthetic data generation. This involves applying transformations to the original images to generate new 'unique' images. Some standard techniques include rotating, flipping, stretching, zooming or brightening. Obviously not all of these make sense depending on the data. In your problem, zooming, stretching and brightening could be used but flipping should not. Rotation could work but only for small angles.

Generally this is implemented by replacing the dataset for each epoch of training. Therefore, the number of images used in each training iteration is the same but the images themselves have been altered.

In practice, it's not a magic bullet. The reason a larger dataset generally yields better models is because the probability of a new feature falling within the feature distribution of the training data is higher. With synthetic data generation the new features are only marginally different to the original so even if the number of images to train on is increased, the feature distributions are not that different. There is a lot of variation in handwritten numbers so it would be very hard to guess how effective this would be without trying it.