Why is this simple neural network not training?

I have created a Tf.Sequential model which outputs 1 for numbers bigger then 5 and 0 otherwise:

const model = tf.sequential();
model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
const xs = tf.tensor2d([[1], [2], [3], [4], [6], [7], [8], [9]]);
const ys = tf.tensor2d([[0], [0], [0], [0], [1], [1], [1], [1]]);
model.fit(xs, ys);
model.predict(xs).print();


With 5 hidden neurons, not even the right trend is detected. Sometimes all the number are too low, or the outputs decrease even if the inputs increase or the outputs are too high.

I later thought that the best way to do this is to have 2 neurons, where 1 is for the input and the other applies a sigmoid function to the input. The weight and bias should easily be adjusted to make the ANN work.

const model = tf.sequential();
model.add(tf.layers.dense({ units: 1, activation: 'sigmoid', inputShape: [1]}));
model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
const xs = tf.tensor2d([[1], [2], [3], [4], [6], [7], [8], [9]]);
const ys = tf.tensor2d([[0], [0], [0], [0], [1], [1], [1], [1]]);
model.fit(xs, ys);
model.predict(xs).print();


Sometimes, this ANN does detect the right trend (the higher the input, the higher the output), but still, the results are never correct and are usually simply too high, always providing an output too close to 1.

How do I make my ANN work, and what have I done wrong?

Edit:

This is the code I'm using now, same problem as before:

const AdadeltaOptimizer = tf.train.adadelta();

const model = tf.sequential();
model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
const xs = tf.tensor1d([1, 2, 3, 4, 5, 6, 7, 8, 9]);
const ys = tf.tensor1d([0, 0, 0, 0, 0, 1, 1, 1, 1]);
model.fit(xs, ys, {
epochs: 2000,
});
model.predict(xs).print();

tf.losses.meanSquaredError(ys, model.predict(xs)).print();


You should try bigger learning rate / more epochs.

What you see is basically output of randomly initialized weights. Default learning rate for SGD is 0.001, it's definitely not enough for weights to change quick enough with single batch epochs and 9 examples.

I never used TF.js, but in keras (that's look very similar) you can set it like this

sgd = keras.optimizers.SGD(lr=1)
model.compile(sgd, 'mse')

model.fit(xs, ys, epochs=200)


Another approach would be to use adaptive learning rate optimizer, for example adadelta. It still requires quite a lot of epochs, but much less, then SGD with standard learning rate and you don't need to set LR manually.

model.compile('adadelta', 'mse')
model.fit(xs, ys, epochs=2000)


Here you can read how different optimizers work and converge

Edit: Another thing with tf.js (and javascript code in general) is it doesn't guarantee order of operations. So, in your case it prints evaluation before it finishes training.

This code should perform fine. Take note that we evaluate model in then clause.

const model = tf.sequential();
model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
const xs = tf.tensor1d([1, 2, 3, 4, 5, 6, 7, 8, 9]);
const ys = tf.tensor1d([0, 0, 0, 0, 0, 1, 1, 1, 1]);
model.fit(xs, ys, {epochs: 1000}).then(h => {
console.log("Loss: " + h.history.loss[0]);
model.predict(xs).print();
});


You can also use async/await approach, for example as it described in this answer

• I tried experimenting with different amounts of epochs, higher learning rates and Adadelta optimizer, but all of it to no avail. I noticed that usually, the amount of not detected trends is approximately also the amount of correctly detected trends for every setting of the network (by setting I mean a certain optimizer and amount of epochs, and for every optimizer and amount of epochs I repeated the training process over and over again to see how the network would perform). Would this suggest that its performance is dependent on its starting position (the randomly chosen weight and bias)? – DaddyMike Aug 13 '19 at 10:59
• Performance indeed depends on network seed, but it would only influence convergency rate for that simple example, because there is no local minimum, only global one. I made keras implementation, it got 100% accuracy for each of 10 runs I did, you can try yourself gist.github.com/mephody-bro/49700ee69c78eabd5af1d6a818fb7ef6. – Kirill Fedyanin Aug 13 '19 at 14:00
• (see edit) Same network, same functions, same amount of epochs... If javascript reads the tensors correctly and the default js batchsize is just the whole data set then I have literally no idea why we are getting different results. – DaddyMike Aug 13 '19 at 14:36
• I've updated answer; It seems you also need to force tf.js execute operations in right order. – Kirill Fedyanin Aug 14 '19 at 15:47