The objective is to design a proximal policy optimization component that has specific constraints on the action space dependent upon state driven rules, using a framework like Tensorforce.
Design Options Listed in the Question
These options are listed here for quick reference when reading the initial analysis below.
- Change the action space at each step, depending on the internal_state. I assume this is nonsense.
- Do nothing : let the model understand that choosing an unavailable action has no impact.
- Do -almost- nothing : impact slightly negatively the reward when the model chooses an unavailable action.
- Help the model : by incorporating an integer into the state/observation space that informs the model what's the internal_state value + bullet point 2 or 3
It is indeed sensible to change the action space for each move. That is, in fact, a proper representation for the problem as stated and the normal way humans play games and the way computers beat humans in Chess and Go.
The apparent senselessness of this idea is merely an artifact of the progress along the Tensorforce project road map and the progress along reinforcement theory, both young in the bigger picture. Reading the Tensorforce documentation and FAQ, it does not appear that the framework is designed to plug in a rules engine to determine the action space. This is not a shortcoming of the open source. There do not appear to be any papers providing theory or proposing algorithms for rule-conditioned Markov chain decisioning.
The do-nothing option is the one that fits into the current available strategies represented in the literature. The do-almost-nothing is probably the approach that will produce more reliable and perhaps more immediate desirable behavior.
The problem with the concept of helping the model is that it is not as strong an idea than extending the model. In open source, this would be done by extending the classes that represent the model, which would require some theoretical work prior to coding to
a. Represent rule-conditioned learning in nomenclature
b. Represent convergence mathematically using the new nomenclature
c. Determining a method of convergence
d. Proving convergence
f. Defining a smooth and efficient algorithm
g. Providing PAC learning information for planning
f. Peer review
g. Extending the classes of the library
h. Proof of concept with the current problem above
i. Additional cases and metrics comparing the approach with the others
j. Extending the library flexibility to support more such dev
The extension of learning systems to cover the rule-constrained case is a great idea for a PhD thesis and might fly in research laboratories as a project proposal with many possible applications. Don't let all the steps dissuade the researcher. They're essentially a list of steps for any PhD thesis or funded AI laboratory project.
For a short term solution, helping the model might work, but it is not a sound strategy for furthering the ideas of AI along the reinforcement learning path. As a short term solution for a particular problem it may work fine. The do-almost-nothing idea may be more sound, since it fits within the convergence proofs that led to the particular implementation Tensorforce is likely to be using.
Renaming it from do-almost-nothing to assist-convergence may help develop the right perspective before giving it a try. You may find that you have to attenuate the assist as you approach convergence to avoid overshoot just as with a learning rate.