I created a custom env to simulate a sort of blocks using gym enviroments.

My environment consists of an 8 x 8 observation space, which would be the stacks of blocks and the height of each stack.

yard: spaces.Box(low=-np.inf, high=np.inf, shape=(8, 8), dtype=float)

The action space is discrete, being mapped to a dictionary that contains all POSSIBLE movements in this environment, which total 168, namely:

self.ACTION_LOOKUP = {0: "(0,1,0) - top slab from stack 0 to top of stack 1",
2: "(0,1,1)" - two
                               1: "(0,2, 2) - top three slabs from post 0 to top of stack 2"}

My reset, in addition to starting the variables that are used throughout the environment, generates a "random map", based on the seed. This map is the arrangement of blocks and location of which must be unlocked in the correct sequence.

Once this is done, the step is designed for 3 main rules

  1. Do not allow an action that is not in space to be performed (this scenario only occurs when the rendering mode is like HUMAN)
  2. Do not allow an action to be performed starting from an empty stack or with fewer blocks than what is being requested.
  3. Do not allow a stack to receive more blocks than its capacity.

In this case the action is not performed and the observation space remains the same.

The reward function, which I have already implemented countless times... is currently designed to capture the general state of the environment and no longer the UNIQUE MOVEMENT (the results are not the same when compared in the evolution of reward_mean). In this case, it is a complex punctuation composition:

  1. If the above validations are checked (of invalid movements, return -1)

Otherwise, I EXECUTE THE CHANGE IN THE STATE OF THE ENVIRONMENT, I calculate the reward (after the movement has been executed), according to the following composition:

  1. 1 point for moving a correct pile (where it contains a block)
  2. Percentage 0 to 100, of how close I can complete unlocking the blocks - this calculation is based on proximity to the surface
  3. Additional points for partially unlocking a block
  4. Reducer (-1) for the current step, my intention is for the game to be completed in the least amount of steps possible.

Dict of my state + image rendered as human

Sad :( - Always decrease

  • $\begingroup$ @NeilSlater thanks, I reviewed my question. Take a look if it makes sense now. $\endgroup$
    – Betnem
    Commented Oct 4, 2023 at 10:57

1 Answer 1


You don't explain how you convert state to neural network features. Which makes me think that you are doing no feature scaling, using the state directly as input. This is a common mistake when implementing deep RL. The neural network can perform badly with inputs outside of range -2 to +2. So one thing you can do is scale the state vector before using it as input to the neural network.

Another thing that might be useful because you have a large number of actions is to look at double DQN, where you run two updating estimators, using one to select the maximising action but the other to generate the TD target from it. This reduces the bias from taking the "best" action from lots of approximations, and then using the very likely overestimated action value from the same estimate.

I can't rule out other sources of error here. Other possibilities include that your state is missing key data that impacts future reward. I'm not sure if that might be the case from your description.

  • $\begingroup$ used this to convert to 1d array def stateDictToArray(self, state): return np.concatenate([ state["yard"].reshape(-1) ]) $\endgroup$
    – Betnem
    Commented Oct 4, 2023 at 13:01
  • $\begingroup$ just add an image to my question. $\endgroup$
    – Betnem
    Commented Oct 4, 2023 at 13:10
  • $\begingroup$ @Betnem If you are using values of 30 or over as inputs to the neural network, my first paragraph definitley applies. This will make the NN perform badly. $\endgroup$ Commented Oct 4, 2023 at 13:52
  • $\begingroup$ thanks a lot. I will make a change in my state to 0 to represent void slots and another value to represent occupied slots... thanks! $\endgroup$
    – Betnem
    Commented Oct 4, 2023 at 14:04
  • $\begingroup$ Didnt get any good results... $\endgroup$
    – Betnem
    Commented Oct 4, 2023 at 14:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .