10
$\begingroup$

I am training a neural network where the target data is a vector of angles in radians (between $0$ and $2\pi$).

I am looking for study material on how to encode this data.

Can you supply me with a book or research paper that covers this topic comprehensively?

$\endgroup$
1

2 Answers 2

17
$\begingroup$

The main problem with simply using the values $\alpha \in [0, 2\pi]$ is that semantically $0 = 2\pi$, but numerically $0$ and $2\pi$ are maximally far apart. A common way to encode this is by a vector of $\sin$ and $\cos$. It perfectly conveys the fact that $0 = 2\pi$, because:

$$ \begin{bmatrix} \sin(0)\\ \cos(0) \end{bmatrix} = \begin{bmatrix} \sin(2\pi)\\ \cos(2\pi) \end{bmatrix} $$

This encoding essentially maps the angle values onto the 2D unit circle. In order to decode this, you can calculate $$\text{atan}2(a_1, a_2) = \alpha,$$

where $a_1 = \sin(\alpha)$ and $a_2 = \cos(\alpha)$.

Here is a nice detailed explanation and here are two references, where this is applied:

EDIT As it was noted in the comments: The values $\sin(\alpha)$ and $\cos(\alpha)$ are not independent and the following naturally holds: $\sqrt{\sin(\alpha)^2 + \cos(\alpha)^2}= 1$, i.e. the euclidean norm is one. In a situation where your Neural Network predicts the sin and cos values, this condition isn't necessarily true. Therefore, you should consider adding a regularization term to the loss that guides the neural network toward outputting valid values (with unit norm) which could look like this:

$$ r_\lambda\left(\hat{y}_1, \hat{y}_2\right)\; = \lambda \left(\; 1 - \sqrt{\hat{y}_1^2 + \hat{y}_2^2}\right), $$

where $\hat{y}_1$ and $\hat{y}_2$ are the sin and cos outputs of the network respectively and $\lambda$ is a scalar that weights the regularization term against the loss. I found this paper where such a regularization term is used (s. Sec. 3.2) to get valid quaternions (Quaternions must also have unit norm). They found that many values work for $\lambda$ and they settle for $\lambda = 0.1$

$\endgroup$
7
  • 1
    $\begingroup$ Do you know if there is a general approach for that? The closest thing that comes to my mind are group convolutional networks. Personally though, I would prefer a simpler, less number crunch heavy method $\endgroup$
    – Imago
    Nov 27, 2022 at 13:12
  • 1
    $\begingroup$ You only need to use $sin(\alpha)$ (or $ccos(\alpha)$) as the label, right? No need (and undesirable) to have both as multilabel outputs given their strict dependence. $\endgroup$ Nov 27, 2022 at 15:23
  • 1
    $\begingroup$ Actually, you should use both otherwise you would get singularities, e.g. for $\sin$ this would happen at $\sin(0) = \sin(\pi) = \sin(2\pi)$. Whereas, if you encode it as the tuple, you get a unique vector for all $\alpha \in [0, 2\pi)$ with the property that at $0$ and $2\pi$ the encoding is the same. You are right that the values are dependent. I've often seen that there is a regularization term in NN training like this: $\parallel [x_1, x_2]^T \parallel_2 = 1$, which guides the network toward outputting valid numbers for the two outputs $x_1, x_2$ where the norm is one. $\endgroup$
    – Chillston
    Nov 27, 2022 at 16:45
  • 1
    $\begingroup$ Thanks for pointing that out @SnehalPatel, I added the regularization thing to the answer $\endgroup$
    – Chillston
    Nov 27, 2022 at 17:02
  • 1
    $\begingroup$ @Chillston I didn't have a specific data set in set. I was wondering how one would handle the case, when the given data has some symmetries, structure or invariances towards specific transformation. In case of angular data as above, a neural network can learn to treat the regions are around an angle 0 and angle 2pi equally, however it is not guaranteed that the network will pick up on that structure. So, how can one simplify this process in general, as done with the angle problem above? $\endgroup$
    – Imago
    Nov 30, 2022 at 14:57
0
$\begingroup$

You might want to look at the von Mises Distribution, it defines a probability distribution over angles.

See Pattern Recognition and Machine Learning, Christopher Bishop, Appendix B (pg 693), or alternatively Wikipedia has an article on this.

You could certainly use this as a loss function in a neural network. My only reservation is that the input parameter $\theta_o$ is itself periodic which might not play well with standard neural network architectures? Therefore previous answers are also worth considering.

I only mention this as if you are interested in distributions over angles, the von Mises distribution is something you should probably be aware of.

$\endgroup$
2
  • 4
    $\begingroup$ I'm not sure I understand what you mean by "You could certainly use this as a loss function in a neural network?" Can you provide an explanation of how it the probability density function can be used as a loss function? $\endgroup$ Nov 28, 2022 at 14:47
  • 1
    $\begingroup$ Von Mises distribution defines a probability density. You take the logarithm of this function and then take the negative of that; that is the loss function. This provides what I think is a pretty good explanation of the basic approach: link $\endgroup$ Nov 29, 2022 at 15:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .