I'm trying to train a RL agent on a custom, highly stochastic environment (MDP). In order to do so I'm using existing implementations of state-of-the-art RL algorithms as provided by Stable Baselines. However, no matter what algorithm I try out and despite extensive hyperparameter tuning, I'm failing to obtain any meaningful result. More precisely, a trivial "always perform the same action (0.7,0.7) each time" strategy works better than any of the obtained policies. The environment is highly stochastic (model of a financial market). How likely is it that the environment is simply "too stochastic" for any meaningful learning to take place? If interested, here's the environment code:
class environment1(gym.Env):
def __init__(self):
self.t = 0.0 # initial time
self.s = 100.0 # initial midprice value
self.T = 1.0 # trading period length
self.sigma = 2 # volatility constant
self.dt = 0.005 # time step
self.q = 0.0 # initial inventory
self.oldq = 0 # initial old inventory
self.x = 0 # initial wealth/cash
self.gamma = 0.1 # risk aversion parameter
self.k = 1.5 # intensity of arivals of orders
self.A = 140 # constant
self.done = False
self.info = []
high = np.array([np.finfo(np.float32).max,
np.finfo(np.float32).max,
np.finfo(np.float32).max],
dtype=np.float32)
self.action_space = spaces.Discrete(100)
self.observation_space = spaces.Box(-high, high, dtype=np.float32)
self.seed()
self.state = None
def seed(self, seed=None):
self.np_random, seed = seeding.np_random(seed)
return [seed]
def step(self, action):
old_x, old_q, old_s = self.x, self.q, self.s # current becomes old
self.t += 0.005 # time increment
P1 = self.dt*self.A*np.exp(-self.k*(action//10)/10) # probabilities of execution
P2 = self.dt*self.A*np.exp(-self.k*(action%10)/10)
if random.random() < P1: # decrease inventory increase cash
self.q -= 1
self.x += self.s + (action//10)/10
if random.random() < P2: # increase inventory decrease cash
self.q += 1
self.x -= self.s - (action%10)/10
if random.random() < 0.5:
self.s += np.sqrt(0.005)*self.sigma
else:
self.s -= np.sqrt(0.005)*self.sigma
self.state = np.array([self.s-100,(self.q-34)/25,(self.t-0.5)/0.29011491975882037])
reward = self.x+self.q*self.s-(self.oldx+self.oldq*self.olds)
if np.isclose(self.t, self.T):
self.done = True
self.oldq = self.q
self.oldx = self.x
self.olds = self.s
return self.state, reward, self.done, {}
def reset(self):
self.t = 0.0 # initial time
self.s = 100.0 # initial midprice value
self.T = 1.0 # trading period length
self.sigma = 2 # volatility constant
self.dt = 0.005 # time step
self.q = 0.0 # initial inventory
self.oldq = 0.0
self.oldx = 0.0
self.olds = 100.0
self.x = 0.0 # initial wealth/cash
self.gamma = 0.1 # risk aversion parameter
self.k = 1.5 # intensity of arivals of orders
self.A = 140 # constant
self.done = False
self.info = []
self.state = np.array([self.s-100,(self.q-34)/25,(self.t-0.5)/0.29011491975882037])
return self.state
The state space is mostly normalized. The action space consists of 100 possible discrete actions (integers from 0 to 99 which are then transformed to (0.0,0.0),(0.0,0.1),...(1.0,1.0). The reward is simply given by the change in the portfolio value (cash+stock).
Note: I've also tried transforming the action space into a continuous one in order to use DDPG but all to no avail.