How did the OpenAI 5 for Dota concatenate units?

Question

I am no expert in the field of AI so I apologize if this is a simple/easy question. I was trying to implement a network similar to OpenAI's for another game and I noticed that I did not fully understand how the network worked.

Below is the image of OpenAI five's network

Basic question

How is the data concatenated/ what is the dimension of the data right after the concatenation before the LSTMs or just before entering the LSTMs? Below are my thoughts which I provide for clarification's sake.

1st Interpretation

In the blue area for units, my initial understanding was that for each visible unit, the output of the max-pool is concatenated along the columns. So, assuming the number of rows is 1(as a 1d max-pool is being applied for each unit), the number of columns is n and there are N visible units, the final size of the matrix when concatenated is $(1,n\cdot N)$ with few extra columns given by the pickups and the like as shown on the left-hand side of the model.

Problem with this interpretation

As the number of units a player can see per each turn is not constant, under this interpretation, I suspect that the fully connected layer after the concatenation layer cannot do its job as matrix multiplication becomes impossible with a variable number of columns.

Possible solution

One possible solution to this is to set a maximum to the number of observed units as $N_{max}$ and pad with constants if some units are not observed. Is this the case?

2nd Interpretation

My 2nd interpretation is that the data is concatenated along the rows. In this case, I can see that the data can pass through a fully connected layer because the number of columns can remain constant. Under this assumption, I decided that right before going through the LSTM, the data is reshaped to (batch size, number of rows, number of columns).

Problems with this interpretation

While I found this interpretation to be more appealing, I noticed that under this train of thought, the LSTM is used just to associate the input data and is not associated with time(the time step for the LSTM is simply the next row of data rather than actual time). I know that this is not especially a problem but I thought that there is no special need to use an LSTM here as in this second interpretation, the order of the data holds no special meaning. But is this the case?

I apologize in advance for any unclear points. Please tell me in the comments and I'll try to clarify as best as I can!

Answer 1

Tl;dr max-pool

You can see in the diagram, everywhere there are a variable number of inputs (pickups, units, hero modifiers/abilities/items), a max-pool follows, though I don't know the specifics of the max-pool implementation.

From https://neuro.cs.ut.ee/the-use-of-embeddings-in-openai-five :

Notice that while the number of modifiers, abilities and items is variable, the network max-pools over each of those lists. This means that only the highest value in all those dimensions actually gets through. At first it does not seem to make sense – it might give the impression that you have an ability that is combination of all existing abilities, e.g. ranged passive heal. But it seems to work for them.

Above processing is done separately for each of the nearby units, the results from general attributes, hero modifiers, abilities and items are all concatenated together. Then different post-processing is applied depending if it was enemy non-hero, allied non-hero, neutral, allied hero or enemy hero.

Finally the results of post-processing are max-pooled over all units of that type. Again this seems to be questionable at the first sight, because different qualities of nearby units would be combined, e.g. if one of the dimensions would represent the health of a unit, then the networks sees only the maximum health over the same type of units. But, again, it seems to work fine.