Dummy variable trap in neural networks and class visualization

Question

Let's say I have data records looking like that: (x1, x2, x3, x4, ..., x100), where each x can be either alpha, gamma or omega.

An example of record could be ('gamma', 'alpha', 'omega', 'alpha', ..., 'gamma') .

So the shape of my dataset is (N, 100) (with N the number of records).

I want to train a neural network to predict some binary label. As there is no underlying ordering in my input categories, I use dummy variables to feed my network. Therefore, I end up with a dataset of the following shape: (N, 100, 3).

My problem is that I don't really know how to deal with the dummy variable trap. According to this answer, I should drop one category when I use a network without weight decay. However, I thought that even without weight decay, the non-linearity of neural networks (assuming I'm using non-linear activation functions like relu), would be enough to avoid the issue without actually needing to drop one category. Am I wrong?

Would a neural network without weight decay behave badly if I do not drop one column?

Some context

Ideally, I would like to avoid dropping one column as my next step is to create inputs that maximize a class prediction (starting from noise and using gradient ascent to "improve" the input).

If I do that with a model trained with a dropped category, I can end up with values close to 0 for my (n-1) categories, probably meaning that the category that would maximize the output would be the dropped one. This interpretation looks correct to me, but it leads to a generated input that has very high-frequency components.

However, I know that consecutive features in a record are likely to be the same (like ('alpha', 'alpha', 'alpha', 'gamma', 'gamma'...)), so input with such high frequencies is quite unrealistic. Generally, this kind of method imposes some L1 or L2 regularization in order to get more global coherency in the generated input. But here is my issue, if I impose some regularisation, the generated input will very likely be biased toward the dropped category right?

As I'm writing these lines, I'm wondering if these considerations about frequencies are not relevant only for numerical inputs and not for categorical inputs. I'm open to any clarification as I'm a bit lost.

Inspiration: The interpretation part of this work is greatly inspired by the first part of this repo. It deals with images as input so the visualization is easier than with my data, but the problem is the same: visualize classes predicted by my network by generating regularized inputs that maximize the class prediction.

EDIT

I believe that the part about using L1 or L2 regularization to improve global coherency is not true. I'm still trying to work out how to solve this problem, but I'm struggling quite a lot.

Answer 1

Assume that you have a (3, 3) dataset (3 records, 3 variables), and a, b, and c are alpha, beta, and gamma, respectively. Then, an example dataframe would be as follows:

   A B C 
1  a b b
2  a a b 
3  g g a 

This dataframe could be one-hot encoded as a (3, 9) dataset. You don't need a (3, 3, 3) dataset as you suggest in the post.

One-hot encoded
   A_a A_b A_g B_a B_b B_g C_a C_b C_g
1  1   0   0   0   1   0   0   1   0
2  1   0   0   1   0   0   0   1   0
3  0   0   1   0   0   1   1   0   0

After you know the value of 2 columns of a feature (e.g., A_a and A_b), you can predict the value of the 3rd column( e.g., A_g). You can, therefore, drop the 3rd (redundant) column for each feature without loss of information, as follows:

One-hot encoded
   A_a A_b B_a B_b C_a C_b
1  1   0   0   1   0   1  
2  1   0   1   0   0   1 
3  0   0   0   0   1   0 

However, the article you refer to suggests to not drop the third column for certain models because "If you remove a category from input of a neural network that employs weight decay, it will get biased in favor of the omitted category instead." Therefore, you should retain the third columns if you employ those algorithms, is what that answer seems to suggest.

For these other models, the answer suggests to drop the third category to avoid multicollinearity:

1. Linear/multilinear regression
2. Logistic regression
3. Discriminant analysis
4. Neural networks that don't employ weight decay

I'm wondering if these considerations about frequencies are not relevant only for numerical inputs and not for categorical inputs.

This issue specifically arises with categorical variables, not numerical variables.