The quote from the paper is:
In this work, we built a custom object detector that provides plausible object candidates.
And in their related submisstion to NeurIPS:
In this work, we built a custom pipeline to provide plausible object candidates. Note that the agent is still required to learn which of these candidates are worth pursuing as goals.
I think ...
Just so that this could be useful for people who refer to this post later on: Please refer to Sutton's reinforcement learning book (2nd edition) example 11.2. It provides an example for why full gradient wouldn't work.
I don't think that (at least from a practical standpoint), there is much difference between continuous action space and discrete action space with >2k discrete actions. Quoting the "Continuous control with Deep RL" paper - which I'd recommend as a starting point for your investigation:
An obvious approach to adapting deep reinforcement learning ...