What is the role of embeddings in a deep recurrent Q network?

Question

When describing the model architecture for a deep recurrent q network, the authors of the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning

each agent consists of a recurrent neural network (RNN), unrolled for $T$ time-steps, that maintains an internal state $h$, an input network for producing a task embedding $z$, and an output network for the Q-values and the messages $m$. The input for agent $a$ is defined as a tuple of $\left(o_{t}^{a}, m_{t-1}^{a^{\prime}}, u_{t-1}^{a}, a\right)$.

Can someone explain what the purpose of the embedding layer is in this specific context?

Implementation can be found here.

Answer 1

The purpose of the input network is to embed the input tuple into a state/task representation, that can then be fed into the RNN hidden state at each time step.

$(o^a_t,m^a′_{t−1},u^a_{t−1},a)$ (input) $\rightarrow$ input network (embedding) $\rightarrow$ $z_t$ (task representation)

---

According to to section 6.1 of the paper, the input is a tuple represented as $(o^a_t,m^a′_{t−1},u^a_{t−1},a)$. Each of these terms is described in sec 3 as:

• $o_t$ - The observation. The authors assume a POMDP

• Two types of actions:

  • $u_t$ - An environment action selects by all the agents at each time step in exchange for a team reward
  • $m_t$ - A communication action, observed by other agent but has no direct impact on the reward or env

• $a$ - The agent (This being a multi-agent DQN algorithm)

They form the input to the Deep Recurrent Q-Network architecture.

The purpose of the embedding network is to receive a tuple of these inputs and produce a state embedding $z$. This state embedding is then fed into a hidden state $h^a_{t-1}$ of the RNN.

Though the authors refer to it as an input network that produces embedding, in practice, they use different embedding functions for each of these inputs. The final task/state embedding $z_t^a$ is expressed as a sum:

$$ z_t^a = \text{MLP}(o_t^a) + \text{MLP}(m_{t-1}) + \text{LookupTable}(u_{t-1}) + \text{LookupTable}(a) $$

The summed embeddings are all of the same size.

This answer makes an assumption that by "embedding layer" you meant the input embedding network. Since the paper makes no reference to a single embedding layer in the model architecture.