Let's say I have data records looking like that:
(x1, x2, x3, x4, ..., x100), where each
x can be either
An example of record could be
('gamma', 'alpha', 'omega', 'alpha', ..., 'gamma') .
So the shape of my dataset is
(N, 100) (with
N the number of records).
I want to train a neural network to predict some binary label. As there is no underlying ordering in my input categories, I use dummy variables to feed my network. Therefore, I end up with a dataset of the following shape:
(N, 100, 3).
My problem is that I don't really know how to deal with the dummy variable trap. According to this answer, I should drop one category when I use a network without weight decay. However, I thought that even without weight decay, the non-linearity of neural networks (assuming I'm using non-linear activation functions like relu), would be enough to avoid the issue without actually needing to drop one category. Am I wrong?
Would a neural network without weight decay behave badly if I do not drop one column?
Ideally, I would like to avoid dropping one column as my next step is to create inputs that maximize a class prediction (starting from noise and using gradient ascent to "improve" the input).
If I do that with a model trained with a dropped category, I can end up with values close to 0 for my (n-1) categories, probably meaning that the category that would maximize the output would be the dropped one. This interpretation looks correct to me, but it leads to a generated input that has very high-frequency components.
However, I know that consecutive features in a record are likely to be the same (like
('alpha', 'alpha', 'alpha', 'gamma', 'gamma'...)), so input with such high frequencies is quite unrealistic. Generally, this kind of method imposes some L1 or L2 regularization in order to get more global coherency in the generated input. But here is my issue, if I impose some regularisation, the generated input will very likely be biased toward the dropped category right?
As I'm writing these lines, I'm wondering if these considerations about frequencies are not relevant only for numerical inputs and not for categorical inputs. I'm open to any clarification as I'm a bit lost.
Inspiration: The interpretation part of this work is greatly inspired by the first part of this repo. It deals with images as input so the visualization is easier than with my data, but the problem is the same: visualize classes predicted by my network by generating regularized inputs that maximize the class prediction.
I believe that the part about using L1 or L2 regularization to improve global coherency is not true. I'm still trying to work out how to solve this problem, but I'm struggling quite a lot.