Which loss / activation function with 2 classes that do not occur often and do not sum to one?

Question

I have a neural network that predicts 2 classes of a time series (bottom and top). Currenlty my Y labels are size 2: [1 0] for bottom and [0 1] for top. The NN has 2 output nodes.

Of course not every instance is a bottom or top. (Overwhelmingly) most of the time we have no bottom and no top.

Should I change my network (and Y labels) to have 3 outputs?

And what would be appropriate as a loss function. I guess with 3 outputs I could use entropy or focal loss. With 2 outputs I don't think entropy is correct.

Currently I use PyTorch (lightening), and I have:

self.lstm = nn.LSTM(input_size=n_features, 
                            hidden_size=hidden_size,
                            num_layers=num_layers, 
                            dropout=dropout, 
                            batch_first=True)
self.linear = nn.Linear(hidden_size, 2)
        


def forward(self, x):
        lstm_out, _ = self.lstm(x)
        y_pred = self.linear(lstm_out[:,-1])
        return y_pred

criterion = nn.CrossEntropyLoss()

Answer 1

This is multi-label classification, which means you have two binary classification problems, one for each of your classes. This is different than multi-class classification.

For this use binary cross-entropy loss, with binary encoded labels (0/1), not one-hot encoded labels, and put a sigmoid activation at the output with two neurons (one for each class).