I am trying to reproduce the recommender task experiment from this paper. The paper suggests to embed discrete actions into continuous action space and then to use the proposed Wolpertinger agent. The Wolpertinger agent is as follows:

DDPG produces so called proto action $f(s)$, then KNN finds k nearest embeddings of discrete actions to this proto action, and then we choose the one of these $k$ embeddings, which has the highest Q-function value. The whole is a full policy, $\pi(\cdot)$.

While training we optimize the critic using only the full policy (DDPG + a choice of a neighbour). The actor is optimized using proto action output in order to be differentiable, $Q(s, f_{\theta}(s)) \rightarrow \max_{\theta}$.

The problem is that the critic does not know that it is used to optimize continuous output of the algorithm. It is trained only to value the embedded actions. As I understand, we hope that continuity of critic will help us with it, but what I have is that proto actions constantly appear to be in some corners with no real actions and where the Q-function unreasonably has greater values (because it is simply untrained in such domains). The DDPG output is normalized to match the embeddings bounds to make these empty spaces not so large.

It seems for me that there is a way to make embeddings more appropriate for the task and achieve higher reward. However, when I use $k = | \mathcal{A}|$, proto actions are not considered and algorithm works pretty well. Usually I use $|\mathcal{A}| = 100$ and $k = 10$. I have trained them with skip-gram, based on the users history.

Below are 2d projections of my embeddings to the first 10 axes (embeddings are from $\mathbb{R}^{20}$). And undependently of the state, proto actions are about the same. The blue is a proto action for the some fixed state. Having some state fixed, $Q(s, f(s))$ value is always higher than $Q(s, a)$ for any $a \in \mathcal{A}$.

enter image description here

Would be glad to get any help, especially the help of people familiar with this algorithm. Do I need to make embeddings fill proto actions range (some hyperrectangle in case we have the tanh activation it the actor)? What is the way to fill such a domain with embeddings?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.