I have a task I want to solve with neural networks. The task is predicting a certain vector of dimension K. The problem is that the inputs to the networks are sparse.
The input is a vector of size N, where N is huge (> 1M) and for most cases, the vast majority of the entries (> 99%) in the input vector are 0. Very rare examples however do have almost full inputs. It's clear that the model is hard to train, as there might be huge weight imbalance within examples, while the target vectors need not have such an imbalance.
I do have a working model, but could you point me to some papers / relevant literature about training a network whose inputs are so sparse? Perhaps there are some techniques that could be useful (maybe some preprocessing steps on the inputs, or something along those lines).
Any hint is appreciated!