6
The most straightforward solution is to simply make every action "legal", but implementing a consistent, deterministic mapping from potentially illegal actions to different legal actions. Whenever the PPO implementation you are using selects an illegal action, you simply replace it with the legal action that it maps to. Your PPO algorithm can then still ...
6
tl:dr
Read chapter 9 of an Introduction of Reinforcement Learning
There is definitely a problem (a curse if you will) when the dimensionality of a task (MDP) grows. For fun, lets extend your problem to a much harder case, continuous variables, and see how we deal with it.
Mood: range [-1, 1] // 1 is Happy, 0 is Neutral, -1 is Sad
Hunger: range [0, 1] //...
4
The answer is "it depends". Once you have arranged the actions into order, a key trait is whether the action value function has a simple enough shape that sampling from a Gaussian policy function would give consistent expected returns, enough that learning can occur. If the underlying "true" value function has a lot of high frequency ...
2
Change the action space at each step, depending on the internal_state. I assume this is nonsense.
Yes, this seems overkill and makes the problem unnecessarily complex, there could be other things you can do.
Do nothing : let the model understand that choosing an unavailable action has no impact.
While this will not harm your model negatively, in any way ...
1
A natural policy to act in an environment with discrete action space would be a softmax.
This paper describes a method that uses the idea of centralized training, and I believe could be used in your implementation.
With regard to your last question, I don't know if i understood, but if you have a system that must perform 3 actions, you could assign each ...
1
Normally, the set of actions that the agent can execute does not change over time, but some actions can become impossible in different states (for example, not every move is possible in any position of the TicTacToe game).
Take a look as example at pice of code https://github.com/haje01/gym-tictactoe/blob/master/examples/base_agent.py :
ava_actions = env....
Only top voted, non community-wiki answers of a minimum length are eligible