1
$\begingroup$

My input to the model is a set of features that I encode in the form of five vectors of the same size consisting only of 0 and 1. I now want to combine them into one vector in such a way that their order does not matter.

My first idea was to run each vector through the same activated linear layer and sum the results.

Is there maybe a better way?

$\endgroup$
2
  • $\begingroup$ Could you give us a concrete example? And what do you mean by "the order of features does not matter", is it that if you permute their order the network's output should be unchanged? $\endgroup$
    – NikoNyrh
    Oct 31, 2022 at 14:04
  • $\begingroup$ @NikoNyrh Simply put, I have to encode a set of sets of features. By set I mean an unordered collection of unique elements. I can encode each inner set as a vector of zeros and ones. Now I need a function that will turn the outer set into a single vector. Simply adding or multiplying these vectors seems to me not to be the right way. $\endgroup$
    – Kasia
    Oct 31, 2022 at 20:58

2 Answers 2

1
$\begingroup$

Is there maybe a better way [to combine them into one vector in such a way that their order does not matter]?

There are many ways to perform feature fusion (see ref). The way you described is one way of doing it. However, for most simple cases where feature fusion is for the type of data you are describing (i.e., tabular data), it is more common to concatenate the features at the input level. The order of concatenation does not matter to the neural network.

$\endgroup$
5
  • $\begingroup$ Could you elaborate on how for a neural network, the order of concatenation does not matter? From the same vectors, depending on the order of concatenation we will get completely different inputs to the network. How could it not matter? $\endgroup$
    – Kasia
    Oct 29, 2022 at 13:41
  • 1
    $\begingroup$ Each of the input features feed into each one of the neurons in the first hidden layer. The inputs are, therefore, all treated the same. Thus, the order of concatenation doesn't matter. This diagram shows how that is true. $\endgroup$ Oct 29, 2022 at 13:55
  • $\begingroup$ I disagree that concatenation would be invariant to the order of the inputs. In the given example the values of h_1 will change if you permute the inputs, assuming that is weights are distinct for each input. $\endgroup$
    – NikoNyrh
    Oct 30, 2022 at 1:08
  • $\begingroup$ The important thing is to be consistent about the order in which the inputs are presented to the model, not the absolute ordering itself. Think about it this way: If you are feeding a neural network two features - petal length and sepal length for example - to predict the iris species, wouldn't it be a bizarre property of the model if it mattered which feature was in column 1 and which was in column 2 of your datatable? Regardless, the order invariance is a directly related to the commutative property of addition. w1*input1 + w2*input2 + b1 = w2*input2 + w1*input1 + b1. $\endgroup$ Oct 30, 2022 at 1:34
  • $\begingroup$ You are right, @NikoNyrh. You cannot change the order randomly. Once you have decided that feature 1 is feeding into input neuron 1, you must be consistent. My point though is that the initial ordering of which feature you decide to be feature 1 which which one you decide to be feature 2 and so on is not considered by the model. $\endgroup$ Oct 30, 2022 at 1:40
0
$\begingroup$

My first idea was to run each vector through the same activated linear layer and sum the results.

That was my first thought as well, but actually it doesn't need to be a single layer but instead it can be a more complex neural network. Basically it will learn an embedding $R^n \rightarrow R^m$, and you can use the sum (or average) of such embeddings to represent the set of your features. I wouldn't use any activation in the final layer, or maybe at most tanh to preserve "symmetry" between negative and positive values. Or who knows, maybe ReLU would work in you problem as well.

I think average of the embeddings would be better than a plain sum, in case the number of "active" features may change and absent features have an embedding of all zeros. Or you normalize the sum so that the L1 or L2 norm is 1.

Come to think of it, the $R^n \rightarrow R^m$ can be implemented as a stack of 1D convolution with "kernel size" and "stride" of one (Keras terms). Or it can be a "stand-alone" model of which outputs you sum together.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .