TF Keras: How to turn this probability-based classifier into single-output-neuron label-based classifier

Question

Here's a simple image classifier implemented in TensorFlow Keras (right click to open in new tab): https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/quickstart/advanced.ipynb

I altered it a bit to fit with my 2-class dataset. And the output layer is:

Dense(2, activation=tf.nn.softmax);

The loss function and optimiser are still the same as in the example in the link above.

loss_fn   = tf.losses.SparseCategoricalCrossentropy();
optimizer = tf.optimizers.Adam();

I wish to turn it into a classifier with single output neuron as I have only 2 classes in dataset, and sigmoid does the 2 classes good. Tried some combinations of output activation functions + loss functions + optimisers, but the network doesn't work any more (ie. it doesn't converge).

For example, this doesn't work:

//output layer
Dense(1, activation=tf.sigmoid);

//loss and optim
loss_fn   = tf.losses.mse;
optimizer = tf.optimizers.Adagrad(1e-1);

Which combination of output activation + loss + optimiser should work for the single-output-neuron model? And generically, which loss functions and optimisers should pair well?

Answer 1

Advice from Neil, yes, output targeting class labels is still classification.

Output range is in contiguous range:

• This is regression
• For example: Linear activation, it has full numeric range of outputs.

Output targeting class labels is classification:

• Single output neuron with sigmoid-like functions. This will classify to 2 classes, although Y data can be normalised to classify more classes.
• Multiple output neurons (probabilities of classes) with sigmoid-like functions (softmax is mainly used). This will classify to 2 or more classes.

Multiple combinations of loss functions and optimisers can make the single-neuron output layer work, with different configs for them. Note that learning rates of different optimisers are different, some take 1e-1, some need 1e-3 for good training.

For example, this combination should work:

loss_fn   = tf.losses.LogCosh();
optimizer = tf.optimizers.RMSprop(1e-3);

From my trying out, these other combinations also work for single output neuron with my data (Adam, Adamax, Nadam, RMSprop work when learning_rate=1e-3 instead of 1e-1):

                         Adadelta  Adagrad  Adam  Adamax  Ftrl  Nadam  RMSprop  SGD
BinaryCrossentropy       Yes       Yes      --    --      --    --     --       Yes
CategoricalCrossentropy  Yes       --       --    --      --    --     --       --
CategoricalHinge         --        --       --    --      --    --     --       --
CosineSimilarity         --        --       --    --      --    --     --       --
Hinge                    Yes       Yes      --    --      --    --     --       Yes
Huber                    Yes       Yes      --    --      --    --     --       Yes
LogCosh                  Yes       Yes      --    --      --    --     --       Yes
Poisson                  Yes       Yes      --    --      --    --     --       Yes
SquaredHinge             Yes       Yes      --    --      --    --     --       Yes

KLD: lambda a,b: KLD(a,b)
MAE,MAPE,MSE,MSLE: lambda a,b: Mxxx(a,b)
The above lambdas are direct functions, not classes like SGD, Adadelta, etc.
SparseCategorialCrossentropy: Seems not working with single output neuron.