A big problem with deep learning methods involving neural networks is that they tend to do really poorly outside the boundaries of the approximation it has learned from the data it is trained on. Unfortunately, because neural networks are black-boxed we have 0 intuitive about where these boundaries might exist. For example, if you train a denoising model to clean words, it's really hard to say what it'll do when it receives a word that has been noised in a manner that it isn't trained on, or if it gets a word with a clean representation it has never seen. Empirically in the cases described in the previous sentence, the model generates something far inferior to what a person can do. I've been examining more conditioned statistical methods for this and wanted some input.

What I've Tried:
A professor I do research under suggested this loss function for a reverse denoising autoencoder. The goal was to noise words in a human approximated way

$P(\theta, \{\hat{x}_i\}|\{y_i\}\{x_i\})\propto {\theta}\mathbf{L_2}(S(\{\hat{x}_i\}), S(\{x_i\}))\prod_i log(p(\theta)p(\hat{x}_i| y_i, \theta))$

With the goal of maximizing $\theta$
$\hat{x_i}$: Single word noised representation of $y_i$ generated by the reverse DAE
$x_i$: Single word noised representation of $y_i$ in data (so I have data of words and what people have noised them too)
$y_i$: Clean word
$S({\cdot})$: Are statistical summaries of the set of data between brackets. These statistics will be put into vector form
$\theta$: Parameters of the DAE
$\mathbf{L_2}$: Euclidean distance or L2 distance

So some of you may be wondering why not just doing standard DAE training since I have the data and use negative log loss. Because when it gets a word that it hasn't seen before as I've mentioned it noises it in a way that is clearly not human. For example, I gave it "Chernovskee" and it noised it to "Teeesovkee", the reason that it is not human approximate is that we ran statistics on the data of clean to noise words and the Levenshtein distance between the two words never exceed 7 and was 1, 65% of the time.

So the Levenshtein statistics I mentioned would go into $S({x_i})$ and that statistic would be taken on $n$ samples of $\hat{x_i}$ and their Euclidean distance is calculated and contributed to the objective function.

The main problem is that I want to minimize the summary statistics between model-generated noised words and the actual data, which is what $\mathbf{L_2}$ is for, but it does not contribute to the gradient at all since there is no $\theta$ and getting stats summaries are non-differentiable operations. So the model has 0 notion about how to backprop to maximize the distance we actually care about.

So to end this long post my question is. Are there methods for calculating summary statistics of model samples that are differentiable OR are there methods for conditioning your model on statistics in a manner that the model has a clear gradient to backprop on?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.