I have an application of neural networks (standard MLP architecture) where I want to forecast a tanh output (ranging from -1 to +1) with about 1500 input features in ~700 samples. Each sample represents a snapshot of database tables at a given time of day - f.ex. at 11:50.
Since I have so few samples, the network is VERY sensitive to overfitting. Although it seems overkill to use neural networks, I find them to perform better than my initial experiments with other approaches, because I can very precisely tune their objective function and regularization, along with a nice possibility for multinomial regression.
A severe problem in the task at hand is that the data in the tables may be adjusted over time. So the database might say that a given feature was last changed at 2016-01-01 11:45, but in reality, a slight change was made a 2016-01-15 07:38. In practice, this means that the training data is unrealistically precise, and that I may instead want to consider it a ballpark estimate. For example, a value of 153.18 may instead be considered a value of ~140-165.
To alleviate the problem, I have theorized a "jitter" neuron that - during training time, at each iteration - "jitters" the input by subtracting or adding a random value from the feature. The random value to add or subtract is a ratio of the standard deviation of the feature, and the ratio is itself random.
The layer is placed right after the input neurons and has as many neurons as there are input features. In my example, the layer is therefore 1500 neurons wide.
For example, consider an input feature that - across all training samples - has a standard deviation of +- 100.
At training time, a data sample is loaded with the feature value of 1122 (for the sake of example, lets ignore input standardization), and the jitter neuron will incur a randomized change in its value.
We define a "jitter" ratio of 0.1, meaning that the ratio to "jitter" the feature with is drawn from a standard distribution with a standard deviation of 0.1 and a mean of 0. In the example, a random roll of the dice lands us in +1.5 standard deviations, outputting a jitter ratio of 0.15.
Given that the training set has a standard deviation of 100, we add to the input feature (that has a value of 1122) a final jitter of +15, resulting in an input feature value of 1137.
The same operation goes for all input features. Pragmatically, I do this in keras by generating a jitter-ratio-matrix with mean=0 and std=0.1. The std-value can be an arbitrary amount, but we must not jitter the input data too much.
The intuitive justification is that it is representative of my real world scenario. Without going into too many details, the input features are typically things such as weather, whose forecasts are naturally unstable, and any changes to the database values backwards in time is likely to be a "typical", small adjustment.
On a more theoretical level, the justification is that it prevents overfitting to specific input features in some samples, as the next iteration across the same data sample will output significantly different output values after the forward pass through the network, as a slight change in the input feature may have a profound impact on the output after the feature has undergone compounding, non-linear activations.
Additionally, I perceive the jitter-neuron to be a form of data-augmentation, like it is done for especialle image classification; instead of the architect generating n augmented samples according to some heuristic (for example, in images, it is normal to crop and rotate the same image in several different ways to enlarge the training set), the jitter neuron generates a theorhetically infinite amount of augmentations at runtime.
I keep imagining the parameter hyperspace for the neuron weights in the first layer; instead of having many potential pits (i.e. areas of overfitting), the jitter smooths out these pits, as the jittered input data now generates a different loss value.
Early experiments do not provide very good results, but not bad either. They just, sort of... Stay the same... However, my dataset is rather small, and it is notoriously difficult to fit with any model.
Therefore, I ask of you what you think of the concept of jitter neurons:
- Does it seem sensible?
- Any theoretical reason that it should/should not work?
- Is it, for some statistical reason or other, inherently inferior to dropout? (notice that I also add dropout, albeit smaller rates)
- Any proposals for making it better?
- Comments, etc.