I'm working in a regression setting to predict a scalar value $y$ from an input $\textbf{x} \in \mathbb{R}^D$ and I'm interested in understanding whenever my model is fed with something that it is outside the (unknown) training distribution $p(\textbf{x})$. For simplicity we can assume I'm using a simple neural network $f_\theta:\mathbb{R}^K \rightarrow\mathbb{R}$ to predict a single (scalar) property value, training my model with an initial dataset $\mathcal{D} = \{(\textbf{x}_i \, , y_i)\}_{i=1}^N$, that is, my task is specifically about regression.

What I'd be interested in achieving would be that, feeding my neural net with a new input $\tilde{\textbf{x}}$ I could retrieve somehow a confidence score telling me if the new input $\tilde{\textbf{x}}$ lies outside the spectrum of observed instances in training dataset.

A way of doing that would be of course estimating the probability of training dataset $p_\theta(\textbf{x})$ and see if the new material $\tilde{\textbf{x}}$ is in a low-likelihood region of $p$. People have used such approach for images (https://arxiv.org/pdf/1912.03263.pdf) but generative models are hard to train.

Instead, I was looking at recently proposed papers using energy-scores for detecting out of distribution samples (paper1, paper2) but the examples seem to refer specifically to classification settings.

As I'm not too familiar with energy-based models, is there a way such frameworks may be applied to regression settings?

  • 1
    $\begingroup$ Though I'm not familiar with energy-based methods, have you tried other "simpler" approaches? for example, assuming a mixture of Gaussians and using Mahalanobis distance as anomaly score, or trying to use kNN on the predicted values of $f_{\theta}$ to find a separating threshold based on mean distance from the neighbors? $\endgroup$ Sep 3, 2022 at 11:13
  • $\begingroup$ Many thanks for your suggestions! As I’m not so into ood I was just searching for what is out there, so I appreciate your comment a lot! Could I ask you to further elaborate on these approaches? $\endgroup$ Sep 3, 2022 at 15:16
  • $\begingroup$ I would suggest going over some of the known AD surveys (here's a pretty new one here - arxiv.org/abs/1901.03407). If you're looking for something shorter you can check out this repo (github.com/Hadar933/Deep-Committee-kNN), try briefing through the introduction section. $\endgroup$ Sep 3, 2022 at 19:03


You must log in to answer this question.