Is it possible, in a transformer or other deep architecture, to include the number of layers as a parameter of the model so it could be learned?
In fact, I have a keras layer that I use to change the final layer without rebuilding the model, so I can just change a parameter between epochs (The original use was to try to train deep networks starting from shallower ones, increasing the number of layers after each epoch).
class LayerSelect(tf.keras.layers.Layer): def __init__(self,nlevels,**kwargs): super(LayerSelect,self).__init__(**kwargs) self.nlevels = nlevels self.range=tf.range(self.nlevels,dtype=tf.float32) def build(self, input_shape): self.kernel=self.add_weight(shape=(1,), initializer=tf.keras.initializers.Constant(min(self.nlevels,14.0)/1.9), trainable=True, dtype=tf.float32, constraint=lambda x: tf.clip_by_value(x,1.0,self.nlevels)) def call(self,inputs): selector=tf.math.maximum([0.0], 1.0 - 1.0 *(self.range-self.kernel)**2 ) final=tf.reduce_sum(inputs*selector,axis=-1) return final
The layer expects an stack of hidden layers to choose from:
allEncoders=tf.stack([encoder[level] for level in range(layers)],axis=-1) finalEncoderRaw=adhoc.LayerSelect(layers)(allEncoders)
So that by calling
set_weights during the training I can choose as output any layer, or a combination of two, being the layer variable a float and using a wider selector, say
1.0 - 0.25 *(self.range-self.kernel)**2
And as you can expect, if I set the weight to be trainable, the optimiser moves the variable. But it keeps either moving randomly some small percent or moving backwards towards smaller values. So it is possible that this approach is a dead end?
If not a way to patch this method, is there another successful method to train the number of layers without using meta-parameter (hyperparameter grids) farms?