Which online machine learning technique to use for multi-class classification problem with multiple inputs?

Question

I have the following problem. We have $4$ separate discrete inputs, which can take any integer value between $-63$ and $63$. The output is also supposed to be a discrete value between $-63$ and $63$. Another constraint is that the solution should allow for online learning with singular values or mini-batches, as the dataset is too big to load all the training data into memory.

I have tried the following method, but the predictions are not good.

I created an MLP or feedforward network with $4$ inputs and $127$ outputs. The inputs are being fed without normalization. The number of hidden layers is $4$ with $[8,16,32,64]$ units in each (respectively). So, essentially, this treats the problem like a sequence classification problem. For training, we feed the non-normalized input along with a one-hot encoded vector for that specific value as output. The inference is done the same way. Finding the hottest output and returning that as the next number in the sequence.

Answer 1

I suggest using Data Stream algorithms to try on your problems since you are asking for "online learning with singular values or minibatches as the dataset is too big too load all the training data into memory."

MOA is a good choice for these algorithms. Hoeffding Trees is also a good first choice to try.

Answer 2

If you prefer to use Python (rather than Java, which is used to implement MOA, which is suggested in the other answer), you might want to have a look at the Python's creme library, whose API is described at https://creme-ml.github.io/api.html, which is a library for incremental and online learning. In particular, you might be interested in the class OneVsRestClassifier.

Answer 3

The way I would approach this (and which I already applied in the past for similar a similar task) is based on multiple points. Let's suppose we're working with PyTorch.

1. I'd start normalizing every input in $[-1, 1]$. Since every feature of the input is in $\mathbb{Z}$ and $\in [-63, +63]$, normalizing it and casting it to fp16 can lower the overall dimension of the batches in the memory, and can allow for automatic mixed precision training. Matter of fact, you should really consider normalizing the values in order to obtain smaller weights in the network.

2. Same thing for the output: Since your task is effectively a bounded regression (those 127 values are not classes), you'd be better off with a single output neuron that can predict values $\in [-1, 1]$ using a tanh activation function. This would rescale the last layer of your network from $Q*H$ (not including biases) to $Q$ weights, where $Q$ is the number of neurons in the last hidden layer and $H$ is the number of output neurons. During inference you can multiply your output by $63$, round it to the nearest integer and obtain your prediction.

3. Since you only have 4 features ranging $[-63, 63]$ you have only $260144641 = 127^4$ values, and it'd be interesting to study the distribution of each single feature, pair of features and triplets of feature, in order to understand if there are some kind of repeated values or collinearities.

4. Usually, earlier layers of the neural networks tend to learn more basic patterns, while deep layers handle more complex, fine details. I'd still suggest, based on my experience and the previous point, to add more neurons to early layers, since you have loads of data (the magnitude of your dataset makes me think it's some sort of scraped values with lots of noise) and lots of updates are gonna hit the weight space in those dimensions. I can sense that the distribution of those features are not uniform, thus you'd avoid handling problems like small differences in input leading to big differences in output with less neurons in the early layers.

5. Yes, the whole dataset can't fit inside the memory, but it can fit inside your disk (if I understood correctly, you are training with a large, offline dataset rather than an online stream of data). There is no need to load it at once though! You can serialize batches of processed data on your disk, and load/unload the batches you need for training, focusing on the biggest batch you can fit in the remaining memory in order to obtain the best results. Of course you're going to account for a time complexity overhead but that was your problem for the get-go. With this solution you will train a neural network without using exotic libraries or frameworks, and focus only on hyperparameter tuning and metrics.