2
$\begingroup$

Given that I'm training a generative model, (say a generative adversarial network), and I know that my (real) inputs (let's say vectors $\textbf{x} \in \mathbb{R}^n$) satisfy linear constraints of the form e.g. $a_1\textbf{x}_1 + \dots a_n\textbf{x}_n =0$, where the coefficients are fixed, is there a way to inject this knowledge during training?

$\endgroup$
4
  • $\begingroup$ are the $a$ coefficient specific values, or generic values? Cause if the latter I suspect the model will collapse and generate only zero values, since that's equivalent to ask the model to solve $XX^{T}=0$ which is true only for $X=0$ $\endgroup$ Mar 8 at 15:10
  • $\begingroup$ also, just as a side note, GANs are not models but rather training approaches, the generative model is only the generator architecture $\endgroup$ Mar 8 at 15:12
  • 1
    $\begingroup$ no, actually the coefficients are thought to be specific, we would assume that the original data lie in the union of hyperplanes for some fixed coefficients. $\endgroup$ Mar 8 at 15:42
  • $\begingroup$ @JamesArten Can you please include these details in the post itself and rewrite the title in the form of a question, i.e. your specific question? Thanks. $\endgroup$
    – nbro
    Mar 8 at 16:29

1 Answer 1

1
$\begingroup$

Maybe a bit too trivial to work out of the shelf, but I would try to add a component to the adversarial loss based precisely on the given set of coefficients.

Something like:

$L_{linear}= \frac{\gamma}{k} \prod_{k}A_{k}\hat{x}$

which combined with the adversarial loss (assuming minimax, but any other choice is fine as well) would become:

$L(G, D)=E_{x}[log(D(x))] + E_{z}[(log(1 - D(G(z)))] + \frac{\gamma}{k} \prod_{k}A_{k}\hat{x} $

where $A_{k}$ is a set of fixed coefficients of an hyperplane, and $\hat{x}$ the vector sampled from the generator. I would use a product operator since the loss should drop to zero when $\hat{x}$ lies in one of the hyperplanes. Instead $\gamma$ can be used to scale the loss to a value in the range of the generator and discriminator losses.

$\endgroup$
4
  • 1
    $\begingroup$ many thanks, that looks interesting. When you say adversarial loss , are you considering the mean loss between generator and discriminator? Are you considering the standard GAN approach where binary classification loss is considered? $\endgroup$ Mar 8 at 16:23
  • 2
    $\begingroup$ precisely, I updated the answer to include the total loss, but I stress out again here that the choice of the adversarial loss doesn't matter, vanilla gan, non saturating gan, wasserstein gan are all good options. And you can add as many extra custom loss to it, as long as you formulate them to be converging to 0. (possibly smoothly, which is what my answer lack, it might be the case that as formulate naively now that extra component will assume large values now and then, fact that doesn't not prevent convergence per se). $\endgroup$ Mar 8 at 16:30
  • $\begingroup$ many thanks Edoardo. Just as a quick clarification, is the constraining term in the loss function intended to be $\frac{\gamma}{k} \mathbb{E}_z[\Pi_k A_k G(z)]$ , so we take the mean over the batch? $\endgroup$ Mar 8 at 17:11
  • $\begingroup$ good point! Yes it's definitely the case that for batches larger than 1 the loss score should be averaged $\endgroup$ Mar 8 at 18:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .