I have a problem where the agent is given an embedding vector to represent the state. Then it is also given a set of possible actions in the environment, let's say that the actions are each represented by a unique text string. The strings don't necessarily follow any natural language rules, though they should contain information about the action.
How does someone use the actor-critic algorithm here?
It is obvious for Q-learning. You just use the $Q(s,a)$ function on each state-action pair. But the actor-network must choose an action directly. And implementations I've seen do this use a categorical output (of all possible actions, which is intractable), or a multi-variate normal distribution of some continuous representation of actions.
I'm not really sure how this applies in this context. And I've considered building a 'fake' actor network out of a Q-network module that just applies the Q-function to each possible action & takes the best action, but I'm not sure if there are some theoretical problems with this which is why the authors of actor-critic avoided the use of a Q-function?