1
$\begingroup$

Hello AI Stack Exchange Community,

I am exploring an idea related to neural networks, and I'm curious to know if this method has been previously researched or if there is a specific term for it.

I am working with two neural networks, A and B:

  1. Network A: This network takes various input data from the environment, and its outputs are used to manipulate the environment in some way.

  2. Network B: The inputs for this network represent both the inputs and outputs of Network A from a given forward pass. The outputs of Network B are meant to represent quantities of interest, such as Nth-order derivatives with respect to time of certain parameters (e.g., robot's battery level or CPU temperature).

Network B is trained in isolation, meaning it is trained separately from Network A. If we are training it to predict Nth-order derivatives, we would need to pre-record the relevant data, and then "replay" it to train Network B.

The combination of Networks A and B results in Network C, where:

  • The output values of A are provided as some of the input values for B.
  • The input values of A are included as the rest of the input values of B.
  • During training, only the weights of Network A are updated, while the weights of Network B remain constant.
  • The target output values for the loss function of Network C can be either manually predetermined or dynamically computed based on the predicted output values.

Both subnetworks A and B have the capability to utilize LSTM layers for handling sequential data.

My main questions for the community are:

  1. Has this method, or something similar, been researched or implemented before?
  2. Is there a specific term or name for this kind of network architecture and training setup?
  3. Are there any obvious challenges or considerations that I should be aware of when implementing this idea?

I am trying to understand if this method has been explored before and if there are any resources or terms that I could look into to learn more about it.

Thank you in advance for your time and assistance!

$\endgroup$
2
  • $\begingroup$ This is similar to reinforcement learning with a secondary network to predict state or observation values in order to better model an environment. However, whether or not it relates to that depends on your goal for this construct. Please explain the motivation behind the structure. As written, network B appears to have no purpose? $\endgroup$ Oct 31, 2023 at 18:28
  • $\begingroup$ @NeilSlater Hi Neil, thank you for the response, and I apologize for the oversight. Network B's purpose is to be differentiable in order to train network A. Network B will most likely output to one neuron, representing the loss of Network A's output. During training, the loss of Network C is "straight-through" (no target/truth outputs or MSE), amounting to be the singular output neuron value of Network B. The weights of Network B are constant, but the weights of Network A are updated during training. $\endgroup$ Oct 31, 2023 at 23:26

1 Answer 1

1
$\begingroup$

Based on my knowledge, there has not been an architecture that covers all the points you have described. Significant development has occurred in creating networks that utilize subnetworks with different tasks for training. Examples include AutoEncoders, GANs, Transformers, Distillation Networks, Siamese Networks, among others. When designing new model architectures, it is crucial to understand the goals for constructing them. For tasks such as detecting an Nth-order derivative with respect to time of a robot's battery level or CPU temperature, there are many methods that could assist, some of which do not even rely on neural networks, like Kalman Filters or ARIMA. If one opts to use neural networks, I would not recommend using anything more complicated than a vanilla Feedforward Network, or for time series problems, an LSTM.

But then, let's try to make the exercise of analyzing this architecture to see if it can be useful for other types of problems.

Network A as a representation function:

We are considering two neural networks. Network A serves as a type of encoder, extracting representations from a given state. This is similar to how an encoder functions in an autoencoder or, to use a reinforcement learning example, how MuZero employs a representation function to distill meaningful data from the state for use by another neural network. We then define Neural Network B, which utilizes this representation to predict a value.

The first point to note is that Network B is designed to take as input both the output of Neural Network A and the original data input. This can be done by concatenating the results or adding them together. This approach raises the question of the purpose of constructing Network A in the first place. If the intent behind Network A was to reduce the dimensionality of the data, why would we then feed it the original data as well? Furthermore, it has been mentioned that Network A would backpropagate the loss using the same loss function as Network B, suggesting that it would attempt to learn the same things as Network B. This is in contrast to, for example, GANs, where two different networks are trained on different loss functions. Consequently, each subnetwork in a GAN learns distinct aspects, providing a rationale to consider them as two separate entities rather than a single large neural network.

I'm not disputing the usefulness of residual connections; they are an essential component in Convolutional Neural Networks and Transformers. However, they are typically employed with a specific purpose in mind, not merely because they are always better.

Freezing the Weights of Network B during the training of Network A:

Freezing weights for fine-tuning pretrained models is a common practice in the field of machine learning today. This method is known as transfer learning, which typically involves taking a pretrained neural network and, to adapt it to a specific task or dataset, adding an extra layer for the output. The pretrained weights are frozen, and the training focuses only on updating this final layer. This technique has proven to be immensely useful across nearly all domains of machine learning, becoming an almost universal strategy for enhancing model accuracy.

What you are describing involves a similar approach, but instead of adding an extra layer at the end, a new network is introduced at the beginning. This raises a question: why do we typically add just a single layer at the end when performing transfer learning? Why not add additional layers or even more complex structures? The answer lies in the operational principles of neural networks and how they update their parameters. A neural network can be thought of as a very large nested function. Within such functions, even minor modifications can significantly impact the network's output.

For instance, consider the difference in the loss function of $x^2 + 1$ and $(x^2 + 1)^2$ compared to $x^2$.If you calculate the average distance between the points of $x^2 + 1$ and $x^2$, the average difference is 1. However, between $(x+1)^2$ and $x^2$, the average difference is 2x + 1, which increases with x. This illustrates how a small change in the function's form—where the "+1" is placed—can lead to a large difference in the output.

enter image description here

Backpropagation

Backpropagation updates the network's weights by first calculating the gradient of the final layer and then propagating this gradient to the preceding layer. However, because the gradient can be between 0 and 1, it can decrease with each layer. This is what is known as the vanishing and exploding gradient effect. This means that the most change occurs in the final layers and the least in the initial layers. Hence, the first layers are the hardest to train as their change is very gradual, which means that the features learned by the first layers are the more general and the final layers are the most specific to the input data. This concept is visualized by the features learned by a convolutional neural network. This is the main reason why models are fine-tuned on the last layer; we do not want to significantly alter the initial layers, as this can easily affect the performance of the entire network.

Therefore, training an entire network with a significant number of layers at the front is not advisable because the results can change drastically, and it will take considerable time to train correctly, especially if freezing the other layers.

Experiment with Mnist

I conducted an experiment using a pretrained vanilla multilayer perceptron to predict the MNIST dataset. I then created two additional networks, one with a new layer at the front and another at the end, while freezing the weights of the pretrained model. The results showed that the network with the extra layer at the end performed slightly better than the pretrained model, while the other one performed worse than the pretrained one. enter image description here

Alternatives and Conclusion

The closest alternative that comes to mind is the use of greedy layer pretraining described in this paper. Overall, with the points I mentioned, I do not believe this method will work efficiently. It will certainly learn something because, in the end, you are performing gradient descent, but it may not do so in an efficient manner; in fact, it might even degrade the model, as my experiment demonstrated. Especially for the problem you described—predicting an nth derivative—Kalman filters or more basic time series models could work much better than a neural network.

$\endgroup$
2
  • 1
    $\begingroup$ Thanks for shout out, but beware comments may be deleted at any time. I really don't require any credit, so feel free to give the advice (understanding the goal is important if you want to play with model architectures) without naming me. We all got our knowledge and ideas from somewhere else, after all. $\endgroup$ Nov 6, 2023 at 16:01
  • $\begingroup$ Thank you for your informative answer. At the time of making my question, I was fumbling with ideas for making a network learn how to maximize paperclip production. I have since ditched it after learning about reinforcement learning though. Very cool stuff. $\endgroup$ Nov 24, 2023 at 23:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .