I am constructing a convolutional variational autoencoder for images, starting out with mnist digits. Typically I would specify convolutional layers in the following way:
input_img = layers.Input(shape=(28,28,1)) conv1 = keras.layers.Conv2D(32, (3,3), strides=2, padding='same', activation='relu')(input_img) conv2 = keras.layers.Conv2D(64, (3,3), strides=2, padding='same', activation='relu')(conv1) ...
However, I would also like to construct a convolutional filter/kernel that is fixed BUT dependent on some content related to the input, which we can call an auxiliary label. This could be a class label or some other piece of relevant information corresponding to the input. For example, for MNIST I can use the class label as auxiliary information and map the digit to a (3,3) kernel and essentially generate a distinct kernel for each digit. This specific filter/kernel is not learned through the network so it is fixed, but it is class dependent. This filter will then be concatenated with the traditional convolutional filters shown above.
input_img = layers.Input(shape=(28,28,1)) conv1 = keras.layers.Conv2D(32, (3,3), strides=2, padding='same', activation='relu')(input_img) # TODO: add a filter/kernel that is fixed (not learned by model) but is class label specific # Not sure how to implement this? # auxiliary_conv = keras.layers.Conv2D(1, (3,3), strides=2, padding='same', activation='relu')(input_img)
I know there are kernel initializers to specify initial weights https://keras.io/api/layers/initializers/, but I'm not sure if this is relevant and if so, how to make this work with a class specific initialization.
In summary, I want a portion of the model's weights to be input content dependent so that some of the trained model's weights vary based on the auxiliary information such as class label, instead of being completely fixed regardless of the input. Is this even possible to achieve in Keras/Tensorflow? I would appreciate any suggestions or examples to get started with implementation.