What is an activity vector in capsule neural networks?

Question

I was reading the paper Dynamic Routing Between Capsules and didn't understand the term "activity vector" in the abstract.

A capsule is a group of neurons whose activity vector represents the instantiation parameters of a specific type of entity such as an object or object part. We use the length of the activity vector to represent the probability that the entity exists and its orientation to represent the instantiation parameters. Active capsules at one level make predictions, via transformation matrices, for the instantiation parameters of higher-level capsules. When multiple predictions agree, a higher level capsule becomes active. We show that a discrimininatively trained, multi-layer capsule system achieves state-of-the-art performance on MNIST and is considerably better than a convolutional net at recognizing highly overlapping digits. To achieve these results we use an iterative routing-by-agreement mechanism: A lower-level capsule prefers to send its output to higher level capsules whose activity vectors have a big scalar product with the prediction coming from the lower-level capsule.

I thought a vector is like an array of data that you are running through the network.

I started working through Andrew Ng's deep learning course, but it's all new and terms go over my head.

Answer 1

In a traditional neural network, the network's vertices are neurons and the output of a single neuron is a single value (a "scalar"). This number is called its activation. A layer of neurons in the network outputs a vector of activations. We should not confuse this with the activity vectors in a Capsule Network.

Capsule Networks are different since the network vertices are Capsules rather than neurons. They are a higher-dimensional: the output of a Capsule is not a scalar but a vector representing a group of parameters related to the input. Hence the name activation vector.

Motivation

In a neural network, there is no inherent structure between the scalar outputs of the neurons, this is something the following layers have to learn. In Capsule Networks the output of a capsule represents all the parameters related to that together in a vector including a prediction for the activation of deeper layer Capsules. This adds a useful local structure.

For example, consider face recognition. If you have a capsule that knows how to recognize eyes it could output an activity vector representing e.g. "since I have recognized an eye position $(x,y)$ with probability $p=0.97$ I predict the parameters for the whole face will be $(f_1,\dots, f_n)$".

As explained in the Dynamic Routing Between Capsules paper you refer to this information is then used in the way that the capsules in earlier layers (the parts: eye, mouth, nose) predict the activations of deeper layers (face). For example, a face recognizer will only be strongly activated when there is an agreement between the eye, nose and mouth recognizers (the parts) and the face recognizer (the whole) about where the face is located (the $(f_1,\dots, f_n)$ parameters).

Historical Inspiration

Older computer vision algorithms like SIFT work in a similar way where recognition is based on agreement between the configuration of multi-dimensional features (key points) and the reference configuration.

Answer 2

I took it to mean something like "the vector of activations of the neurons in the capsule". The activation for a given neuron is the weighted sum of its inputs, passed through the activation function (sigmoid, relu, etc).