I'm working on a project lately and I'm trying to solve a problem with reinforcement learning and I have serious issues with shaping the reward function.
The problem is designing a device with maximum efficiency. So we simulated the problem as follows. There is a 4x4 grid (we defined a 4x4 matrix) and the elements of this matrix can be either 0 or 1 (value 0 means "air" and 1 means a certain material in reality), so there is a 2^16 possible configurations for this matrix. Our agent starts from the top left corner of this matrix and has 5 possible actions: move up, down, left, right and flip (which means flipping a 0 to 1 or vice versa). Based on flipping action, we get a new configuration and each configuration has an efficiency (which is calculated by maxwell equations in the background).
Our goal is to find the best configuration so that the efficiency of the device is maximum.
So far we have tried many reward functions and non of them seemed to work at all! I will mention some of them:
reward = current_efficiency - previous_efficiency (the efficiency is being calculated in each time step)
if current_efficiency > previous_efficiency: reward = current_efficiency previous_efficiency = current_efficiency
diff = current_efficiency - previous_efficiency if diff > 0: reward = 1 else: reward = -2
and some other variations. Nothing is working for our problem and the agent doesn't learn at all! So far, we have used different approaches to DQN and also A2C method and so far no positive feedback. We tried different definitions of states as well, but we don't think that is the problem.
So, can somebody maybe help me with this? It would be a huge help!