How is the constraint $\|f(x)\|_{2}=1$ enforced for the embedding $f(x)$ in the FaceNet paper?

Question

In the FaceNet paper, under section 3.2, the authors mention that:

The embedding is represented by $f(x) \in \mathbb{R}^{d}$. It embeds an image $x$ into a $d$-dimensional Euclidean space. Additionally, we constrain this embedding to live on the $d$ dimensional hypersphere, i.e. $\|f(x)\|_{2}=1$.

I don't quite understand how the above equation holds. As far as I understand, the $L_2$ norm is the same as the Euclidean distance, but I don't quite understand how this imposes $\|f(x)\|_{2}=1$ criteria.

Answer 1

The constraint is enforced with bespoke code. If FaceNet was implemented in NumPy, and the embedding layer vector (pre-constraint) was in the NumPy array h, then the code might look like:

e = h / np.linalg.norm(h)

The variable e would then contain the desired embedding with L2 distance of 1. In practice, even with NumPy, the code might be more complex due to handling mini-batches. More likely this will not be implemented in NumPy.

In this Keras/TensorFlow-based FaceNet implementation you can see how it may be done in practice:

# L2 normalization
X = Lambda(lambda  x: K.l2_normalize(x,axis=1))(X)

This scaling transformation is considered part of the neural network code (it is part of the Keras model building routine in the above snippet), so there needs to be corresponding support for back propagation through the embedding. When using an automatic differentiation framework such as Theano or TensorFlow, then no extra code is required.