Is my backpropagation code correct? [closed]

I am trying to implement the back-propagation algorithm for the following neural network.

2 input -> 3 input layer -> 1 output
Activation f(x) -> sigmoid
Loss f(x) -> Yexpected-Yresult


This is the Matlab/Octave function for backpropagating an XOR ANN. However, it doesn't seem to work, and it converges to 0.5. Is it correct?

sigmoid = @(z) 1./(1 + exp(-z));
sig_der = @(y) sigmoid(y).*(1-sigmoid(y));
function [w1, w2, b1, b2] = backprop(w1,w2,b1,b2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i)
delta2 = (sig_der(last_layer,sigmoid)).*cost;
delta1 = (sig_der(mid_layer,sigmoid)).*sum(delta2.*w2);
w2 = w2 + 0.001 .* mid_layer .* delta2;
w1 = w1 + 0.001 .* data(1:2,i) .* delta1.';
b1 = b1 + 0.001 .* delta1';
b2 = b2 + 0.001 .* delta2;
end


Here is the code:

clear

graphics_toolkit("gnuplot")
sigmoid = @(z) 1./(1 + exp(-z));
sig_der = @(y) sigmoid(y).*(1-sigmoid(y));

function [cost, mid_layer, last_layer] = forward(w1,w2,b1,b2,data,sigmoid,i)
mid_layer = sigmoid(sum(data(1:2,i).*w1)'-b1);
last_layer = sigmoid(sum(mid_layer.*w2)-b2);
cost = data(3,i)-last_layer;
end

function [w1, w2, b1, b2] = backprop(w1,w2,b1,b2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i)
delta2 = (sig_der(last_layer,sigmoid)).*cost;
delta1 = (sig_der(mid_layer,sigmoid)).*sum(delta2.*w2);
w2 = w2 + 0.00001 .* mid_layer .* delta2;
w1 = w1 + 0.00001 .* data(1:2,i) .* delta1';
b1 = b1 + 0.00001 .* delta1;
b2 = b2 + 0.00001 .* delta2;
end

data(:,1)=[0; 0; 0];
data(:,2)=[1; 0; 1];
data(:,3)=[0; 1; 1];
data(:,4)=[1; 1; 0];

w1=rand(2,3)./2.*(rand(2,3).*-2+1);
w2=rand(3,1)./2.*(rand(3,1).*-2+1);
b1=rand(1,3)./2;
b2=rand(1,1)./2;

for j=1:20000
for i=1:4
[cost, mid_layer, last_layer] = forward(w1,w2,b1,b2,data,sigmoid,i);
[w1, w2, b1, b2] = backprop(w1,w2,b1,b2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i);
cost_mem(j,i)=cost;
end
end
toc

• Note that general programming issues or bugs are not on-topic here. However, I was hesitant to close this post because the issue could lie in your understanding of the back-propagation algorithm, which is on-topic. See ai.stackexchange.com/help/on-topic for more details. – nbro Apr 2 '20 at 21:15
• Thanks. You can close it if you want or let other people comment on it if they feel like it ... – david david Apr 2 '20 at 21:22
• Even if this post is closed, people can still comment on it, but they cannot provide an answer anymore, which makes sense, because you solved your issue. – nbro Apr 2 '20 at 21:26

Thanks for everyone's help. I have now solved the problem. I hadn't quite understood the back propagation algorithm. I invite people to take a look at this link: backpropagation @ AGH UST which has solved my problem.

Code:

clear
graphics_toolkit("gnuplot")
tic
sigmoid = @(z) 1./(1 + exp(-z));
sig_der = @(y) sigmoid(y).*(1-sigmoid(y));

function [cost, mid_layer, last_layer] = forward(w1,w2,data,sigmoid,i)
mid_layer(:,1)=sum(w1.*data(1:2,i));
mid_layer(:,2)=sigmoid(mid_layer(:,1));
last_layer(:,1)=sum(mid_layer(:,2).*w2);
last_layer(:,2)=sigmoid(last_layer(:,1));
cost = data(3,i)-last_layer(:,2);
end

function [w1, w2] = backprop(w1,w2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i)
delta(1) = cost;
delta(2:4) = cost .* w2;
w2 = w2 + 0.99 .* delta(1) .* mid_layer(:,2) .* sig_der(last_layer(:,1));
w1 = w1 + 0.99 .* delta(2:4) .* sig_der(mid_layer(:,1))' .* data(1:2,i);
%b2 = b2 + 0.01 .*
%b1 = b1 + 0.01 .*
end

tic

data(:,1)=[0; 0; 0];
data(:,2)=[1; 0; 1];
data(:,3)=[0; 1; 1];
data(:,4)=[1; 1; 0];

w1=rand(2,3)./2.*(rand(2,3).*-2+1);
w2=rand(3,1)./2.*(rand(3,1).*-2+1);
%b1=ones(1,3);%rand(1,3)./2;
%b2=ones(1,1);%rand(1,1)./2;

for j=1:10000
for i=[randperm(4)]
[cost, mid_layer, last_layer] = forward(w1,w2,data,sigmoid,i);
[w1, w2] = backprop(w1,w2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i);
cost_mem(j,i)=cost;
end
end

toc

%{
dlmwrite("/tmp/weights_1",w1);
dlmwrite("/tmp/weights_2",w2);
%}