I'm new to playing around with deep learning. I'm trying to find a function that satisfies a bunch of constraints, and looking for tips on how to better my approach.

Let $F:X\times Y \mapsto [0,1]$, where $X=[0,Z]$,$Y=[0,M]$ compact. $F$ is a cdf over $X$, and twice differentiable in both arguments. I'm trying to find such an $F$ that satisfies, well that it's a cdf, but also a few more constraints that can be expressed locally. Most constraints have to do with monotonicity of the function, or it's derivatives, or some transformation of those derivatives. There is one last constraint (call it $g^*$) which I'm most interested in, and is supposed to hold only at a single point $x^*,y^*$.

There is no labelled data to be fitted here - I'm just searching over the space of functions $F$, evaluate them at their arguments, and seeing if they satisfy all constraints.

My naive approach was to try a fully connected deep neural net, where I'm playing around with nr of parameters, layers, type of activation functions. My loss just computes the penalties for constraint violations, sums them up and weights them - here also maybe points of improvement.

My loss function vaguely looks like this: $$L(x,y,NN) = mon(x,y,NN) + intg(x,y,NN) + MLRP(x,y,NN) + g^*(x,y,NN), $$ where $mon,intg,MLRP$ are monotonicity constraints, constraints that ensure $NN$ outputs a cdf, and monotonicity on some kinds of growths rates of the pdf, and finally $g^*$. These constraints are all expressed as inequality constraints. Even though $g^*$ is supposed to hold only at the point $x^*,y^*$, I was playing around with a sort of kernelized approach, applying a weighted down version of $g^*$ for points nearby the critical points. Didn't help.

The issue is, after a while of trying different combinations of the above (I'm mostly using Adam, also played around with parameters there), I can't seem to get to satisfy both $g^*$ and all other constraints. I can tell that the main issue is $g^*$ - I can find functions that satisfy the other more `natural' constraints, but adding $g^*$ doesn't get me far. I had thought that something flexible with many parameters would be able to force the satisfaction of many kinds of single point constraint, but alas. Before I give up and accept that maybe this problem doesn't actually have a solution, I came here to ask whether I'm doing the wrong approach.

A strange thing that commonly happens during training is that $g^*$ is close to being satisfied sometime during the first few iterations, but the penalty converges to something far away. Here a sample plot of the $g^*$ violation over training:

enter image description here

What I'm asking is for some tips/insights as to how to generally approach such a problem - maybe fully connected neural nets is very wrong here? Also intuitions for why the process doesn't seem to like to stay at those `better satisfaction' points around periods 30 and 40?



You must log in to answer this question.