14
$\begingroup$

Given a neural network $f$ that takes as input $n$ data points: $x_1, \dots, x_n$. We say $f$ is permutation invariant if

$$f(x_1 ... x_n) = f(pi(x_1 ... x_n))$$

for any permutation $pi$.

Could someone recommend a starting point (article, example, or other paper) for permutation invariant neural networks?

$\endgroup$
  • $\begingroup$ Interesting question. You want to treat you inputs as a set rather than a vector. Assuming your inputs are scalar or otherwise comparable, did you consider sorting your inputs to create a permutation-invariant canonical representation and feeding that to a regular network? $\endgroup$ – mjul Dec 4 '17 at 15:15
  • $\begingroup$ @mjul My inputs are series of bets of clients (each bet is represented by a vector of some cathegorical and continuous variables). I could order them chronologically for example, but as the time spacing between bets for each client is very different, this wouldn't make much sense. This could probably be solved using some function (fixed or learned) of time that would decay the coefficients for each bet. But I think the ordering really doesn't matter in this case so I wanted to try unordered data first, which obviously requires to treat the bets for each client symmetrically. $\endgroup$ – Josef Ondrej Dec 5 '17 at 8:33
  • $\begingroup$ What is the network output? If the network is permutation invariant, the output will be the same for any order of inputs. Is this what you want? $\endgroup$ – BlueMoon93 Dec 7 '17 at 17:22
  • $\begingroup$ @BlueMoon93 Yes, that is exactly, what I want. The output can be anything (number, vector) as long as it does not depend on order of inputs. $\endgroup$ – Josef Ondrej Dec 10 '17 at 19:44
  • $\begingroup$ why would you want a permutation invariant neural network? $\endgroup$ – k.c. sayz 'k.c sayz' Dec 12 '17 at 2:57
2
$\begingroup$

As far as I know, no one has tried this, due to the way the network is structured. Each input has a set of weights, that are connected to more inputs. If the inputs switch, the output will too.

However, you can build a network that approaches this behaviour. In your training set, use batch learning and for each training sample, give all possible permutations to the network such that it learns to be permutation invariant. This will never be exactly invariant, it just might be close.

Another way to do this is to have the weights replicated for all inputs. For example, lets assume you have 3 inputs (i0, i1, i2), and the next hidden layer has 2 nodes (hl0, hl1) and activation function F. Assuming a fully connected layer, you have 2 weights w0 and w1. The hidden layer's nodes hl0 and hl1 are given, respectively, by

  • hl0 = F(i0w0+i1w0+i2w0)

  • hl1 = F(i0w1+i1w1+i2w1)

Thus giving you a hidden layer whose values are permutation invariant from the input. From now on, you can learn and build the rest of the network as you see fit. This is an approach derived from convolutional layers.


Off-topic, this seems like a cool project. If you want to cooperate on some research project, contact me (check my profile)

$\endgroup$
  • $\begingroup$ The first approach suggested would be infeasible in my case due to the computational complexity. The second on the other hand seems maybe too restrictive. But it is certainly a good start. What I have come up with so far is approach similar to the one I found in this paper: arxiv.org/pdf/1612.04530.pdf. First I consider all pairs (generally all k-tuples) of inputs x_i, x_j, i,j in 0 ... n and apply some neural network on all of them (the same nn on each pair). This gives me n**2 outputs f(x_i, x_j) and then I average them (or take maximum) and apply anoter nn over the result. $\endgroup$ – Josef Ondrej Dec 11 '17 at 20:16
  • $\begingroup$ This is what I have come up with so far: github.com/josefondrej/Symmetric-Layers $\endgroup$ – Josef Ondrej Jan 31 '18 at 14:02
5
$\begingroup$

Here is a few that might be what you are looking for:

$\endgroup$
3
$\begingroup$

I have implemented Permutational Layer here using Keras: https://github.com/off99555/superkeras/blob/master/permutational_layer.py

You can call the PermutationalModule function to use it.

Implemented following this paper: https://arxiv.org/pdf/1612.04530.pdf

The idea is to compare all pairs of N^2 pairs from N inputs, use the model with shared weights, then use pooling function N times on N inputs. The output you can use pooling again but in the paper, they don't mention another pooling.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.