0
$\begingroup$

I come from a math background, so I am not up-to-date with machine learning literature.

For the purpose of learning dynamics, I would like to train a model to minimize the following loss: $$\mathcal{L} = |\mathcal{M}(\mathcal{M}(x)) - y|,$$ more generally, there can be $k \geq 2$ recursive applications of the model $\mathcal{M}$.

From my experimentation, regular fully-connected neural networks perform poorly—a single application of $\mathcal{M}$ performs better and intermediate results do not correspond to intermediate data.

Are there other architectures that are better suited for this problem? Maybe graph-based models or the U-net from DDPMs?

Edit: Just a note on RNNs (I could be completely wrong): Although RNNs are used for recursive applications, I don't think they are what I am looking for. RNNs accept a sequence of inputs, whereas my input is a single tensor. Also, going from one hidden state to the next corresponds to a matrix multiplication which seems not very expressive(?).

Improve this question
$\endgroup$
1
  • $\begingroup$ In StyleGAN there is two steps mapping and synthesis. Since the mapping step has same amount of inputs as outputs, I once had an idea to run it twice using its own output as input. And let's just say the results can be interesting sometimes if you do something you do not have any idea about the outcome. $\endgroup$
    – harism
    Jul 22 at 0:50

1 Answer 1

Reset to default
1
$\begingroup$

You mentioned dynamics, so I assume you have some kind of vectorised state $S$ and the model should output $S'$ in the same space. A transformer will do this.

A special case of UNet which outputs the same size tensor as its input would also work, if your input is image-like (amenable to convolution).

An RNN would also work, but its recursive architecture wouldn't be the reason.

A neural network is essentially just a function, and they have very flexible domains and codomains.

If you want $M(M(x))$ to be well defined, you just need $M$ to map $X \rightarrow X$ which basically any neural network can do. The best architecture mostly depends on $X$.

Improve this answer
$\endgroup$
13
  • $\begingroup$ In my experiments (Lorenz system), a fully-connected NN completely fails. What stops the model from mapping inputs in the $X$ domain to the $Y$ domain and inputs in the $Y$ domain to the $X$ domain? $\endgroup$
    – user572780
    Jul 19 at 15:04
  • $\begingroup$ @user572780 - depends on how you are using "domain", I think the answer here is focused on dimensionality only. And you cannot write a neural network that converts between different dimensionalities by itself automatically e.g. if it receives $\mathbb{R}^n$ it should output in $\mathbb{R}^m$ and vice-versa . . . although you could add supporting code for this and use the same neural network to attempt both tasks with redundancy in unused inputs or outputs depending which step was being processed. $\endgroup$
    – Neil Slater
    Jul 19 at 16:07
  • $\begingroup$ @NeilSlater By "domain" I mean an open connected set—my question was not to do with dimensionality. $\endgroup$
    – user572780
    Jul 19 at 16:22
  • $\begingroup$ @user572780 Then you may have a problem in a single model if you need specific mappings and X/Y domains overlap. But if you don't care how X/Y mapping works, or if the domains do not overlap, then a single NN model should be fine. Or, as a special case, if where X and Y domains overlap, then the mapping is always the exact same vector in X and Y. $\endgroup$
    – Neil Slater
    Jul 19 at 16:35
  • $\begingroup$ @NeilSlater I think there was a misunderstanding. Essentially, I am applying a NN twice so that the intermediate results follow the dynamics of the problem. It seems that a fully-connected NN cannot do this. $\endgroup$
    – user572780
    Jul 19 at 17:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .