# Combining neural network outputs via matrix multiplication

I'm replicating a particular network architecture which is very sparse with its details. One part of said architecture is shown in the image, where h is a 1x1024 or 1024x1 output from a transformer and needs to be combined with the output of an MLP to produce a 1x55 or 55x1 vector, with the order of the dimensions not mattering. As such the matrix multiplication step should involve either multiplying (55 x k) and (k x 1), or (1 x k) and (k x 55) matrices. The input to the MLP is a (55 x 32) matrix flattened to a (1760 x 1) vector. For context, this input encodes item information from a game about 55 items. h contains information about the current game state and we want to produce a policy pi for selecting an item.

The constraints of the matrix multiplication mean that I need to perform some kind of dimensionality increase or reduction on either the elements of h or the output of the MLP. However, I'm not really sure what would be best in this case as I don't have much experience with building neural networks in general. Any advice would be appreciated.

## 2 Answers

You can try padding, which compensates the dimension with numbers (e.g., zero). This practice is very common in deep learning to have a desire embedding.

• To improve this answer add some explanatory text about what padding is. Links can break or become invalid so it's best to include enough information to make the answer self-contained and durable Commented Jan 31, 2023 at 22:40

In my view, the strange part seems to be the flattening of the useable_items matrix. At the very least, it is confusing for the reader of the article. If we want an output vector pi where the values actually correspond to each of the 55 input items, the information of which input belongs to which output should be kept.

So either:

• It should be reshaped back to (55 x k) after the MLP.
That would make every value for every item depend on each other in the MLP
Also, this approach makes it hard to keep track of the input/output relationship
• Or the matrix should be passed through the MLP as is, only using the features of the last dimension.
This effortlessly gives you a (55 x k) matrix where you could matmul away to get the desired output
And easily keep the input/output relationship
..and keep the number of MLP parameters way down.

In either case you should probably choose an output dimension in the MLP to either directly get (or reshape to) (55 x 1024) as output dimension and treat the matmul as a dot product over the last dimensions to get your (55 x 1) vector.

• ...and I just realized this question was from February 2023, not 2024. So, hopes up someone may still find it useful. Commented Mar 20 at 9:03