17
$\begingroup$

Given a neural network $f$ that takes as input $n$ data points: $x_1, \dots, x_n$. We say $f$ is permutation invariant if

$$f(x_1 ... x_n) = f(\sigma(x_1 ... x_n))$$

for any permutation $\sigma$.

Could someone recommend a starting point (article, example, or other paper) for permutation invariant neural networks?

$\endgroup$
  • 1
    $\begingroup$ Interesting question. You want to treat you inputs as a set rather than a vector. Assuming your inputs are scalar or otherwise comparable, did you consider sorting your inputs to create a permutation-invariant canonical representation and feeding that to a regular network? $\endgroup$ – mjul Dec 4 '17 at 15:15
  • 1
    $\begingroup$ @mjul My inputs are series of bets of clients (each bet is represented by a vector of some cathegorical and continuous variables). I could order them chronologically for example, but as the time spacing between bets for each client is very different, this wouldn't make much sense. This could probably be solved using some function (fixed or learned) of time that would decay the coefficients for each bet. But I think the ordering really doesn't matter in this case so I wanted to try unordered data first, which obviously requires to treat the bets for each client symmetrically. $\endgroup$ – Josef Ondrej Dec 5 '17 at 8:33
  • 1
    $\begingroup$ What is the network output? If the network is permutation invariant, the output will be the same for any order of inputs. Is this what you want? $\endgroup$ – BlueMoon93 Dec 7 '17 at 17:22
  • 1
    $\begingroup$ @BlueMoon93 Yes, that is exactly, what I want. The output can be anything (number, vector) as long as it does not depend on order of inputs. $\endgroup$ – Josef Ondrej Dec 10 '17 at 19:44
  • $\begingroup$ why would you want a permutation invariant neural network? $\endgroup$ – k.c. sayz 'k.c sayz' Dec 12 '17 at 2:57
7
$\begingroup$

Traditionally,, due to the way the network is structured, each input has a set of weights, that are connected to more inputs. If the inputs switch, the output will too.

However, you can build a network that approaches this behaviour. In your training set, use batch learning and for each training sample, give all possible permutations to the network such that it learns to be permutation invariant. This will never be exactly invariant, it just might be close.

Another way to do this is to have the weights replicated for all inputs. For example, lets assume you have 3 inputs (i0, i1, i2), and the next hidden layer has 2 nodes (hl0, hl1) and activation function F. Assuming a fully connected layer, you have 2 weights w0 and w1. The hidden layer's nodes hl0 and hl1 are given, respectively, by

  • hl0 = F(i0w0+i1w0+i2w0)

  • hl1 = F(i0w1+i1w1+i2w1)

Thus giving you a hidden layer whose values are permutation invariant from the input. From now on, you can learn and build the rest of the network as you see fit. This is an approach derived from convolutional layers.

Finally, you can use a dimension reduction to achieve this. I've published this in "Exploring Communication Protocols and Centralized Critics in Multi-Agent Deep Learning", on Integrated Computer-Aided Engineering. This approach uses convolutional architectures to efficiently achieve permutation invariance.

| improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ The first approach suggested would be infeasible in my case due to the computational complexity. The second on the other hand seems maybe too restrictive. But it is certainly a good start. What I have come up with so far is approach similar to the one I found in this paper: arxiv.org/pdf/1612.04530.pdf. First I consider all pairs (generally all k-tuples) of inputs x_i, x_j, i,j in 0 ... n and apply some neural network on all of them (the same nn on each pair). This gives me n**2 outputs f(x_i, x_j) and then I average them (or take maximum) and apply anoter nn over the result. $\endgroup$ – Josef Ondrej Dec 11 '17 at 20:16
  • $\begingroup$ This is what I have come up with so far: github.com/josefondrej/Symmetric-Layers $\endgroup$ – Josef Ondrej Jan 31 '18 at 14:02
9
$\begingroup$

Here is a few that might be what you are looking for:

| improve this answer | |
$\endgroup$
5
$\begingroup$

I have implemented Permutational Layer here using Keras: https://github.com/off99555/superkeras/blob/master/permutational_layer.py

You can call the PermutationalModule function to use it.

Implemented following this paper: https://arxiv.org/pdf/1612.04530.pdf

The idea is to compare all pairs of N^2 pairs from N inputs, use the model with shared weights, then use pooling function N times on N inputs. The output you can use pooling again but in the paper, they don't mention another pooling.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Interesting. Without looking into details, I wonder why the computation is $N^2$ instead of $N!$. There are $N!$ possible permutations in the symmetric group. $\endgroup$ – Yan King Yin May 17 at 0:24
  • 1
    $\begingroup$ Because this is not a complete permutation. It's very different from what you think. So you better look at the details from the paper. $\endgroup$ – off99555 May 17 at 3:21
1
$\begingroup$

So, a practical application of this with a lot of research is in the deep lidar processing community. In that world, you have to do a lot of computation on point clouds which are completely unordered. One of the seminal works in this field is Point Net (https://arxiv.org/pdf/1612.00593.pdf) which solves this problem by performing all symmetric operations across channels. 1D convolutions, max pool, etc. Any network in this style that does not project to a 2D representation has to adhere to this rule.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.