Using features extracted from a CNN as convolutional filter

Question

I'm a bit confused about this. Assume I have a CNN network with two branches:

1. Top
2. Bottom

The top branch outputs a feature vector of shape 1x1x1x10 (batch, h, w, c) The bottom branch outputs a feature vector of shape (1, 10, 10, 10).

I want to use the top feature vector as a convolutional filter, and convolve it with the bottom feature vector. I can do this in pytorch with the "functional.Conv2D" function, the problem is, I don't know how back-prop works in this case (will it be unstable?) since the output feature is a now a parameter as well, do I need to stop gradients or do something else in this case to backprop correctly?