1
$\begingroup$

I am writing an MDP based agent that is supposed to learn to place bids and asks in a trading environment. The system requests 2 values (mWh energy and $, both being positive or negative). Every timestep the agent has a certain volume that it has to either buy or sell.

I tried setting these two values as action values, giving it 4 individual ones (1 for buy price and amount one sell price and amount)

I used the DDPG and NAF agents from keras-rl here but both aren't working for me. I tried a number of reward functions too:

  • direct cash reward: average price of market for required energy vs what the agent achieved
  • shifting balancing price: first emphasize that the broker balances it's portfolio (i.e. orders the amount it has to) and later optimize for price per mWh
  • simple core: as a test I ran a reward function that just rewards the agent to be close to the actions [0.5, 0.55]

All three failed again.

  • LR : tried between 0.01 and 0.00001
  • Layers: Tried anything between 1 layer 1 cell and 5 layer 128 cells
  • Types: I used both Dense and LSTM cells with according input shapes

Symptoms: Generally it looks like the system is not learning anything. I am unsure why. How does the reward function have to be structures to incentivize the system to at least move in the correct direction? Especially the reward that told the agent to be close to [0.5, 0.5] by basing the reward simply on the squared difference to this point should have worked in my eyes.

enter image description here

$\endgroup$
  • 2
    $\begingroup$ I recommend you do sanity checks by setting extrema rewards. For example in a buy-sell scenario I would set the buying price to 0 and the selling price to 1, in which case the agent should learn to buy everything. And similarly if the selling price is 0 and the buying price is 1 it should learn not to buy anything. Make your problem very very simple in the beginning and add the complex parts incrementally. $\endgroup$ – Hai Nguyen May 29 '18 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.