13 events
when toggle format what by license comment
Oct 15 '21 at 12:32 comment added PeterBe Thanks Neil for your answers. I accepted and upvoted your answer.
Oct 13 '21 at 12:52 comment added Neil Slater In general, RL will cope better with soft constraints for state values, with negative rewards, but it can also cope OK with failure/early termination as a solution for hard constraints. You also asked about avoiding termination - my answers there are with regard to hard constraints only.
Oct 13 '21 at 12:50 comment added Neil Slater @PeterBe Yes that's fine if it is a soft constraint and that state can actually occur physically in the real system. My answer about excluding states from training is regarding strict constraints. Your question includes examples ranging from soft constraints to strict/hard constraints, so I have done my best to cover both in the answer. Only you can say when looking at a constraint whether it is a hard or soft constraint for the system
Oct 13 '21 at 12:22 comment added PeterBe Thanks Neil for your answer. Actually the control system always begins at the beginning of the day and the states there are all set to valid values. So the starting states are always valid states in my example. Thus it is not possible to start the training from unreachable states. I don't really understand the second part of your comment. The training always beginns with valid states but it can happen that the states violate some constraints. For this I use a penalty such that the agent gets bad rewards when beeing in such a state. If the agent is in such a state it should try to get out of it
Oct 13 '21 at 12:10 comment added Neil Slater For (2) by not training your system on the unreachable states as starting states. You have already decided a constraint that some state is not reachable, and you can detect this (otherwise you would not be able to calculate any penalty for reaching that state). So skip memorising and/or training for any (s,a) pair where s is outside your constraints. You will still want to train for (s,a,r,s') wheer s' is unreachable in orderto apply the constraint penalty and stop the agent going there.
Oct 13 '21 at 12:00 comment added PeterBe Thanks Neil for your answer and effort. I really appreciate it. Regarding (3) I have aske a separate question (ai.stackexchange.com/questions/32042/…). Bur regarding (2) I still do not understand your answer. How can I make sure "that if a state is not reachable, learning a policy decision for it may be a waste of time." that this is not happening?
Oct 12 '21 at 16:57 comment added Neil Slater For (2) the assumption is that you have a state space constraint. So for certain don't train on data that starts from unreachable states according to your constraint. For (3) I think that would be a good separate question on the site. You can also choose to interpret the lower figure as zero regardless in terms of effect. I don't know whether that would be better than your options for remodelling the action space to force only valid actions.
Oct 12 '21 at 13:14 comment added PeterBe Thanks Neil for your answer and effort. I really appreciate it. Regarding your answer to 2) " I mean that if a state is not reachable, learning a policy decision for it may be a waste of time." --> how can I make sure that this is not happening? Regarding your answer to 3) "the point is not to adjust things afterwards, but to construct the action space so that only valid actions are possible." --> how can I do that? I have two continuous variables x(t) and y(t). Now IF x(t) is >0 THEN y(t) has to be 0 and vice versa with x(t) and y(t) element 0 or [0.25,1]. How can tell this to the agent?
Oct 11 '21 at 20:13 comment added Neil Slater @PeterBe: For (1) Yes that's what I mean. For (2) I mean that if a state is not reachable, learning a policy decision for it may be a waste of time. For (3) I don't understand your comment, but I think you are saying that you have a continuous action vector, or at the least multiple settings for each heater. In either case this is not two actions, but more - potentially a lot more. However, you can still expand on the idea in the answer - the point is not to adjust things afterwards, but to construct the action space so that only valid actions are possible.
Oct 11 '21 at 15:50 comment added PeterBe 3) About the absolute rules I have not 3 but 2 actions (one for each story). Can I just directly change the actions in each iteration of the learning such that this constraint is not violated? For example if the RL agent wanted to assign a positive value for each 2 actions (which is not possible) I would just overwrite these proposed actions from RL with my own rule? Where would I have to do this? In the step function of the environment or shall I rather define a new separate control class that can overrule the actions of the RL agent.
Oct 11 '21 at 15:50 comment added PeterBe Thanks a lot Neil for your answer. I have 3 questions on that. 1) About Hard constraints you wrote "terminate the episode early and score 0 for breaking the constraint." --> What do you mean by terminating the episode? To finish the RL and give a reward of 0? I think in my case this would happen many times such that the learning might suffer from that. 2) About the hard constraints you wrote "That is because the agent learning how to "escape" from an unreachable state may not be useful." --> What do you mean by that?
Sep 24 '21 at 18:56 history edited Neil Slater CC BY-SA 4.0
added 62 characters in body
Sep 24 '21 at 18:50 history answered Neil Slater CC BY-SA 4.0