Why is this simple neural network not training?

Question

I have created a Tf.Sequential model which outputs 1 for numbers bigger then 5 and 0 otherwise:

const model = tf.sequential();
model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
model.add(tf.layers.dense({ units: 1, activation: 'sigmoid'}));
model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
const xs = tf.tensor2d([[1], [2], [3], [4], [6], [7], [8], [9]]);
const ys = tf.tensor2d([[0], [0], [0], [0], [1], [1], [1], [1]]);
model.fit(xs, ys);
model.predict(xs).print();

With 5 hidden neurons, not even the right trend is detected. Sometimes all the number are too low, or the outputs decrease even if the inputs increase or the outputs are too high.

I later thought that the best way to do this is to have 2 neurons, where 1 is for the input and the other applies a sigmoid function to the input. The weight and bias should easily be adjusted to make the ANN work.

const model = tf.sequential();
model.add(tf.layers.dense({ units: 1, activation: 'sigmoid', inputShape: [1]}));
model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
const xs = tf.tensor2d([[1], [2], [3], [4], [6], [7], [8], [9]]);
const ys = tf.tensor2d([[0], [0], [0], [0], [1], [1], [1], [1]]);
model.fit(xs, ys);
model.predict(xs).print();

Sometimes, this ANN does detect the right trend (the higher the input, the higher the output), but still, the results are never correct and are usually simply too high, always providing an output too close to 1.

How do I make my ANN work, and what have I done wrong?

Edit:

This is the code I'm using now, same problem as before:

const AdadeltaOptimizer = tf.train.adadelta();

const model = tf.sequential();
model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
model.add(tf.layers.dense({ units: 1, activation: 'sigmoid'}));
model.compile({loss: 'meanSquaredError', optimizer: AdadeltaOptimizer});
const xs = tf.tensor1d([1, 2, 3, 4, 5, 6, 7, 8, 9]);
const ys = tf.tensor1d([0, 0, 0, 0, 0, 1, 1, 1, 1]);
model.fit(xs, ys, {
epochs: 2000,
});
model.predict(xs).print();

tf.losses.meanSquaredError(ys, model.predict(xs)).print();

Answer 1

You should try bigger learning rate / more epochs.

What you see is basically output of randomly initialized weights. Default learning rate for SGD is 0.001, it's definitely not enough for weights to change quick enough with single batch epochs and 9 examples.

I never used TF.js, but in keras (that's look very similar) you can set it like this

sgd = keras.optimizers.SGD(lr=1)
model.compile(sgd, 'mse')

model.fit(xs, ys, epochs=200)

Another approach would be to use adaptive learning rate optimizer, for example adadelta. It still requires quite a lot of epochs, but much less, then SGD with standard learning rate and you don't need to set LR manually.

model.compile('adadelta', 'mse')
model.fit(xs, ys, epochs=2000)

Here you can read how different optimizers work and converge

Edit: Another thing with tf.js (and javascript code in general) is it doesn't guarantee order of operations. So, in your case it prints evaluation before it finishes training.

This code should perform fine. Take note that we evaluate model in then clause.

const model = tf.sequential();
model.add(tf.layers.dense({ units: 5, activation: 'sigmoid', inputShape: [1]}));
model.add(tf.layers.dense({ units: 1, activation: 'sigmoid'}));
model.compile({loss: 'meanSquaredError', optimizer: 'adadelta'});
const xs = tf.tensor1d([1, 2, 3, 4, 5, 6, 7, 8, 9]);
const ys = tf.tensor1d([0, 0, 0, 0, 0, 1, 1, 1, 1]);
model.fit(xs, ys, {epochs: 1000}).then(h => {
   console.log("Loss: " + h.history.loss[0]);
   model.predict(xs).print();
});

You can also use async/await approach, for example as it described in this answer