I am writing an MDP based agent that is supposed to learn to place bids and asks in a trading environment. The system requests 2 values (mWh energy and $, both being positive or negative). Every timestep the agent has a certain volume that it has to either buy or sell.
I tried setting these two values as action values, giving it 4 individual ones (1 for buy price and amount one sell price and amount)
I used the DDPG and NAF agents from keras-rl here but both aren't working for me. I tried a number of reward functions too:
- direct cash reward: average price of market for required energy vs what the agent achieved
- shifting balancing price: first emphasize that the broker balances it's portfolio (i.e. orders the amount it has to) and later optimize for price per mWh
- simple core: as a test I ran a reward function that just rewards the agent to be close to the actions [0.5, 0.55]
All three failed again.
- LR : tried between 0.01 and 0.00001
- Layers: Tried anything between 1 layer 1 cell and 5 layer 128 cells
- Types: I used both Dense and LSTM cells with according input shapes
Symptoms: Generally it looks like the system is not learning anything. I am unsure why. How does the reward function have to be structures to incentivize the system to at least move in the correct direction? Especially the reward that told the agent to be close to [0.5, 0.5] by basing the reward simply on the squared difference to this point should have worked in my eyes.