I'm working on a project lately and I'm trying to solve a problem with reinforcement learning and I have serious issues with shaping the reward function.

The problem is designing a device with maximum efficiency. So we simulated the problem as follows. There is a 4x4 grid (we defined a 4x4 matrix) and the elements of this matrix can be either 0 or 1 (value 0 means "air" and 1 means a certain material in reality), so there is a 2^16 possible configurations for this matrix. Our agent starts from the top left corner of this matrix and has 5 possible actions: move up, down, left, right and flip (which means flipping a 0 to 1 or vice versa). Based on flipping action, we get a new configuration and each configuration has an efficiency (which is calculated by maxwell equations in the background).

Our goal is to find the best configuration so that the efficiency of the device is maximum.

So far we have tried many reward functions and non of them seemed to work at all! I will mention some of them:

  1. reward = current_efficiency - previous_efficiency (the efficiency is being calculated in each time step)

  2.  if current_efficiency > previous_efficiency:
        reward = current_efficiency
        previous_efficiency = current_efficiency
  3.  diff = current_efficiency - previous_efficiency
     if diff > 0:
        reward = 1
        reward = -2

and some other variations. Nothing is working for our problem and the agent doesn't learn at all! So far, we have used different approaches to DQN and also A2C method and so far no positive feedback. We tried different definitions of states as well, but we don't think that is the problem.

So, can somebody maybe help me with this? It would be a huge help!

  • 2
    $\begingroup$ how expensive is it to calculate the efficiency? 2^16 doesn't seem like a huge number, could you not just brute force it? other than that, what are you defining as a state, which position in the grid the agent is in? $\endgroup$ – David Ireland Jun 17 at 21:25
  • 2
    $\begingroup$ To find the best configuration seems more like a search rather than a reinforcement learning one ? $\endgroup$ – calveeen Jun 18 at 2:03
  • $\begingroup$ @DavidIreland The problem is that this is a test case. My actual Problem has 10 ^120 possible states. My state consist of the configurations of the grid ( a matirx of 0s and 1s) and another matrix which has all the values = 0 except where currently the agent is. There the value =1. Then these two are flatend and added together. $\endgroup$ – Hossein Jun 22 at 9:20
  • $\begingroup$ @calveeen yes, but we want to find the best policy with reinforcement learning. Thats the whole idea about the project. $\endgroup$ – Hossein Jun 22 at 9:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.