How to train a neural network with few weights and biases held constant?

Question

I am a beginner in neural networks. I am building a neural network with 3 layers. The input $X$ has 7 features and the output $Y$ is a real number. In the hidden layer, there are two nodes. The bottom node contains weights and biases which should be hard set.

Now, I want to train this neural network with the training data $X$ and $Y$, such that the red weights are held constant while all other weights are learnable.

Is there a way of doing this during the training of the neural network? I'm using TensorFlow and Keras, so, if you could provide also the code necessary to do this, that would be very useful.

Answer 1

The strategy is turning b2 into a separate model, initialise b2 the way it should be, and train your network without b2 as usual.

In the middle of the main network, combine the output of b1 layer and b2 network using concatenation function, for example in TensorFlow:

# Axis 0 is the batch dimension, axis 1 for the dimension of value in every sample
tf.concat([b1out,b2out],axis=1)

Example source code (paste to Google Colab to test):

%tensorflow_version 2.x
%reset -f

import tensorflow                   as     tf
from   tensorflow.keras             import *
from   tensorflow.keras.layers      import *
from   tensorflow.keras.activations import *
from   tensorflow.keras.models      import *
from   tensorflow.keras.callbacks   import *

class MyModel(Model):
    def __init__(self):
        super(MyModel,self).__init__()
        self.dense1 = Dense(200, activation=relu)
        self.dense2 = Dense(1,   activation=tf.identity)

    @tf.function
    def call(self,x):
        # MIND THE CONCATENATION IN THIS PART:
        h1a = self.dense1(x)
        h1b = b2(x) # b2 won't get trained, so it stays fixed

        # CONCATENATION AT AXIS 1, COZ AXIS 0 IS BATCH DIMENSION
        h1  = tf.concat([h1a,h1b],axis=1) 
        u   = self.dense2(h1)
        return u

class ModelB2(Model):
    def __init__(self):
        super(ModelB2,self).__init__()
        self.dense1 = Dense(200)
        # self.dense2 = ...
        # Init weights of B here or B is pre-trained model

    @tf.function
    def call(self,x):
        u = self.dense1(x)
        return u

# PROGRAMME ENTRY POINT ========================================================
if __name__=="__main__":
    inp = [[1,2,3,4,5,6,7],[7,6,5,4,3,2,1]] # Example values
    exp = [[0],            [1]            ]

    mm = MyModel()
    b2 = ModelB2()

    mm.compile(loss=tf.losses.MeanSquaredError(), optimizer=tf.optimizers.Adam(1e-3))
    b2.compile(loss=tf.losses.MeanSquaredError(), optimizer=tf.optimizers.Adam(1e-3))
    mm_loss = mm.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1) # Init weights in here
    b2_loss = b2.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1) # Init weights in here

    print("\nbefore training:")
    print("mm weights:",mm.get_weights()[0][0][:3],"...")
    print("b2 weights:",b2.get_weights()[0][0][:3],"...")
    print("mm loss:",mm_loss)
    print("b2 loss:",b2_loss)

    mm.fit(x=inp,y=exp, batch_size=len(inp), epochs=500, verbose=0)
    mm_loss = mm.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1) 
    b2_loss = b2.evaluate(x=inp,y=exp, batch_size=len(inp), steps=1) 

    print("\nafter training:")
    print("mm weights:",mm.get_weights()[0][0][:3],"...")
    print("b2 weights:",b2.get_weights()[0][0][:3],"... <-- UNCHANGED AS WANTED")
    print("mm loss:",mm_loss)
    print("b2 loss:",b2_loss,"<-- UNCHANGED AS WANTED")
# EOF

Google colab: https://colab.research.google.com