0
$\begingroup$

In BERT like transformer model (I am not talking about BERT in this thread), it has 2 training objectives Masked Language Modeling and Next sentence prediction right? and BERT model is also supports different input shapes, So I am actually building a model with 2 training objectives on a base model and those 2 training objectives are Denoising data on time-series and Triplet loss on time sereis and just want to take the base model body and fine tune the model on different data with different shape in TensorFlow, How is code written for this in tensorflow at low-level and what I mean is extract that body add new inputs and new outputs to this base model and fine-tune whole model on some dataset with different shape.

Base Model Arcitechture:

inputs=layers.Input((3000,7))
x = layers.Conv1D(32, 3, activation=tf.nn.leaky_relu,padding='same')(inputs)
x = layers.MaxPooling1D(2)(x)
x = layers.MultiHeadAttention(num_heads=4, key_dim=2)(x, x)

x = layers.BatchNormalization()(x)

x = layers.Conv1D(64, 3, activation=tf.nn.leaky_relu,padding='same')(x)
x = layers.MaxPooling1D(2)(x)
x = layers.MultiHeadAttention(num_heads=4, key_dim=2)(x, x)

x = layers.GlobalAveragePooling1D()(x)
x = layers.Dense(512, activation='relu')(x)
x = layers.Dense(256, activation='relu')(x)
outputs = layers.Dense(128, activation='relu')(x)


base_model = keras.Model(inputs=inputs, outputs=x)

Pre-training Base model:

input_denoising = keras.Input(shape=(x_unlabelled.shape[1], x_unlabelled.shape[2]))
x = base_model(input_denoising)
x=layers.Dense(3000*7,activation=tf.nn.leaky_relu)(x)
output_denoising=layers.Reshape((3000, 7))(x)

input_1 = keras.Input(shape=(x_unlabelled.shape[1], x_unlabelled.shape[2]))
input_2 = keras.Input(shape=(x_unlabelled.shape[1], x_unlabelled.shape[2]))
input_3 = keras.Input(shape=(x_unlabelled.shape[1], x_unlabelled.shape[2]))

output_1 = base_model(input_1)
output_2 = base_model(input_2)
output_3 = base_model(input_3)

concatenated = layers.concatenate([output_1, output_2, output_3])
output = layers.Dense(2, activation='softmax')(concatenated)


combined_model=keras.Model(inputs=[input_denoising,input_1, input_2, input_3],outputs=[output_denoising,output])
combined_model.compile(optimizer='adam',loss=['mse','categorical_crossentropy'])

combined_model.summary()

So, now I want to remove all the input and outputs and add new inputs and outputs aligned with new data and fine-tune on that dataset, so how can I do this

$\endgroup$

0

You must log in to answer this question.