Here's a simple image classifier implemented in TensorFlow Keras (right click to open in new tab): https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/quickstart/advanced.ipynb

I altered it a bit to fit with my 2-class dataset. And the output layer is:

Dense(2, activation=tf.nn.softmax);

The loss function and optimiser are still the same as in the example in the link above.

loss_fn   = tf.losses.SparseCategoricalCrossentropy();
optimizer = tf.optimizers.Adam();

I wish to turn it into a classifier with single output neuron as I have only 2 classes in dataset, and sigmoid does the 2 classes good. Tried some combinations of output activation functions + loss functions + optimisers, but the network doesn't work any more (ie. it doesn't converge).

For example, this doesn't work:

//output layer
Dense(1, activation=tf.sigmoid);

//loss and optim
loss_fn   = tf.losses.mse;
optimizer = tf.optimizers.Adagrad(1e-1);

Which combination of output activation + loss + optimiser should work for the single-output-neuron model? And generically, which loss functions and optimisers should pair well?


Advice from Neil, yes, output targeting class labels is still classification.

Output range is in contiguous range:

  • This is regression
  • For example: Linear activation, it has full numeric range of outputs.

Output targeting class labels is classification:

  • Single output neuron with sigmoid-like functions. This will classify to 2 classes, although Y data can be normalised to classify more classes.
  • Multiple output neurons (probabilities of classes) with sigmoid-like functions (softmax is mainly used). This will classify to 2 or more classes.

Multiple combinations of loss functions and optimisers can make the single-neuron output layer work, with different configs for them. Note that learning rates of different optimisers are different, some take 1e-1, some need 1e-3 for good training.

For example, this combination should work:

loss_fn   = tf.losses.LogCosh();
optimizer = tf.optimizers.RMSprop(1e-3);

From my trying out, these other combinations also work for single output neuron with my data (Adam, Adamax, Nadam, RMSprop work when learning_rate=1e-3 instead of 1e-1):

                         Adadelta  Adagrad  Adam  Adamax  Ftrl  Nadam  RMSprop  SGD
BinaryCrossentropy       Yes       Yes      --    --      --    --     --       Yes
CategoricalCrossentropy  Yes       --       --    --      --    --     --       --
CategoricalHinge         --        --       --    --      --    --     --       --
CosineSimilarity         --        --       --    --      --    --     --       --
Hinge                    Yes       Yes      --    --      --    --     --       Yes
Huber                    Yes       Yes      --    --      --    --     --       Yes
LogCosh                  Yes       Yes      --    --      --    --     --       Yes
Poisson                  Yes       Yes      --    --      --    --     --       Yes
SquaredHinge             Yes       Yes      --    --      --    --     --       Yes

KLD: lambda a,b: KLD(a,b)
MAE,MAPE,MSE,MSLE: lambda a,b: Mxxx(a,b)
The above lambdas are direct functions, not classes like SGD, Adadelta, etc.
SparseCategorialCrossentropy: Seems not working with single output neuron.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.