4
$\begingroup$

I am interested in learning about the inverse of neural networks and I would like to understand about the invertibility of neural networks, as for example described in On the Invertibility of Invertible Neural Networks.

Researchers who are working on this domain, can you help me understand these two questions.

  1. Are all neural network invertible ?
  2. What exactly qualifies a neural network to be invertible ?
Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

The meaning of invertible here is the standard definition of invertibility for a mathematical function $f \colon X \to Y$. Invertible simply means "the function has an inverse map $f^{-1} \colon Y \to X$". Equivalently the function $f$ is bijective, which means the following two conditions hold:

  1. $f$ is injective: for any two distinct $x_1, x_2 \in X$, $f(x_1) \ne f(x_2)$.

  2. $f$ is surjective: for any $y \in Y$, there exists an $x \in X$ such that $f(x) = y$.

If this is unfamiliar, you should be able to find some helpful references on Google using these terms.

Most obvious neural network architectures cannot possibly be invertible. Consider for example a classifier which takes an image or some other high-dimensional input, and outputs a classification label. This network could only be invertible if there was only one possible input1 which corresponds to each label, which is not the goal of the network.

1 I mean this rather literally: if one pixel is even slightly different, this would be a different input to the network and would have to have a different output.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you so much that explains it wonderfully! The last point you referred to is exactly what I have been investigating how to invert a high dimensional input for a classification problem. Is there any literature you can refer me to which states that in such cases the invertibility is not possible? $\endgroup$
    – user157522
    Apr 20, 2021 at 9:23
  • 1
    $\begingroup$ @user157522 Glad to hear that. That fact follows from the definition of invertibility, so it might not be stated explicitly in the literature in the context of neural networks. You can probably prove it to yourself after thinking about the definitions (perhaps take a read through the Wikipedia article as a start). In particular there is no way to form an injective map from an infinite set to a finite one (you can find this in any text on set theory, or search for this result directly), which more or less settles the question for a classifier. $\endgroup$
    – htl
    Apr 20, 2021 at 9:32
  • 1
    $\begingroup$ @user157522 Okay, it might be mentioned in passing in a paper on invertible networks, but I don't think the result will be particularly prominent unfortunately. Depending on your audience, assuming a knowledge of elementary concepts like bijections might be reasonable (although of course what is elementary to a mathematician who works with these concepts all day might not be elementary to someone else!). Perhaps giving a brief proof of the result would be sufficient, with references to a mathematical text for readers who would be interested? $\endgroup$
    – htl
    Apr 20, 2021 at 10:21
  • 1
    $\begingroup$ The answer is technically incorrect in the sense that neural nets generally output a real number (which is actually what I want). The rest of it is true. You might want to change that. $\endgroup$
    – user9947
    Apr 20, 2021 at 14:37
  • 1
    $\begingroup$ Excellent answer. For @user157522, you may be interested to check out hci.iwr.uni-heidelberg.de/vislearn/… and github.com/jaekookang/invertible_neural_networks. And a recent example from my field: science.org/doi/10.1126/science.aaw1147 $\endgroup$
    – R. Barrett
    Sep 29, 2022 at 22:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .