1
$\begingroup$

I am working image reconstruction project. It is a part of multispectral image fusion. I am referring paper in the link mentioned below.

paper link: https://arxiv.org/pdf/2101.09643v1.pdf

For image reconstruction, authors have used a network which looks like this: enter image description here

And after implementing this, for training, and changing parameters, the loss (difference between reconstructed and actual image) is calculated.

In the image above, after extraction of two feature sets, authors are simply giving these feature sets to the decoder network after concatenation. I was wondering, can there be any better way than just concatenating these two feature sets, like weighted sum maybe ? For combining these feature sets?

I will really appreciate if anyone can suggest any ideas.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

There are various methods to combine two sets of features (say $x$ and $y$):

  • Concatenation (as done in figure): done along a given dimension (usually last axis), to obtain new features $z=[x\ y]$. The resulting $z$, will have a dimension that is $dim(z) = dim(x)+dim(y)$. The advantage of this method is that is flexible, meaning the next layers should understand how to make good use of the extra channels or dimensions. Disadvantage: the result is larger (more dimensions).
  • Addition: just element-wise add the two features $z=x+y$. Advantage: same num. of dimensions, but less flexible.
  • Multiplication: element-wise multiply the two features $z=x\odot y$. Same pros and cons of the addition strategy.

Now, it's not possible a priori to determine which of these is better. You should just try each of them and see how it performs.

A weighted sum can be a good approach especially if $x$ and $y$ have some meaning:

  • $z = a\cdot x + b\cdot y$. The coefficients $a$ and $b$ can be set manually, learned (1-dimensional scaling factors, or have the same dim. of the features), or set like $b=(1-a)$.
  • If you learn $a$ and $b$ you may want to bound them in [0,1] via sigmoid or softmax (if multi-dimensional): this is known as gating, if I remember well.
  • In principle you can even combine addition with multiplication.

An even more sophisticated approach would be to have a small sub-network with two inputs and one output, to learn how to best combine $x$ and $y$ in a non-linear way.

The last tip is that if you have some domain knowledge on the problem and data, you may be able to introduce that when designing the operation that combines the two feature sets.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Anyalone Thanks for the inputs. I think I will stay away from parameters that I will have to further learn in the model. Because the goal is to achieve min possible training time. So, less the parameters, the better. $\endgroup$
    – programmer_04_03
    Commented Apr 18, 2023 at 20:16
  • $\begingroup$ And I was also thinking about some convolution operations with predefined kernels such as Random Gaussian or average value. Do you think that will be helpful? $\endgroup$
    – programmer_04_03
    Commented Apr 18, 2023 at 20:26
  • $\begingroup$ In general, when you want to combine two feature sets you want to also combine (and so preserve) their respective information. I guess introducing random kernels is a bad idea, maybe the average is a little better. Anyway, in the paper section D (page 4) they propose 2 strategies: addition and channel strategy. Have you looked at the channel strategy? Maybe you can combine (or weight) the RGB features with the pooled ones of the multispectral input $\endgroup$
    – Luca Anzalone
    Commented Apr 19, 2023 at 9:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .