1
$\begingroup$

I'm trying to train a RL agent on a custom, highly stochastic environment (MDP). In order to do so I'm using existing implementations of state-of-the-art RL algorithms as provided by Stable Baselines. However, no matter what algorithm I try out and despite extensive hyperparameter tuning, I'm failing to obtain any meaningful result. More precisely, a trivial "always perform the same action (0.7,0.7) each time" strategy works better than any of the obtained policies. The environment is highly stochastic (model of a financial market). How likely is it that the environment is simply "too stochastic" for any meaningful learning to take place? If interested, here's the environment code:

class environment1(gym.Env):
def __init__(self):
    self.t = 0.0 # initial time
    self.s = 100.0 # initial midprice value
    self.T = 1.0 # trading period length
    self.sigma = 2 # volatility constant
    self.dt = 0.005 # time step
    self.q = 0.0 # initial inventory
    self.oldq = 0 # initial old inventory
    self.x = 0 # initial wealth/cash
    self.gamma = 0.1 # risk aversion parameter
    self.k = 1.5 # intensity of arivals of orders
    self.A = 140 # constant
    self.done = False
    self.info = []
    high = np.array([np.finfo(np.float32).max,
                     np.finfo(np.float32).max,
                     np.finfo(np.float32).max],
                    dtype=np.float32)
    self.action_space = spaces.Discrete(100)
    self.observation_space = spaces.Box(-high, high, dtype=np.float32)
    self.seed()
    self.state = None

def seed(self, seed=None):
    self.np_random, seed = seeding.np_random(seed)
    return [seed]

def step(self, action):
    old_x, old_q, old_s = self.x, self.q, self.s # current becomes old
    self.t += 0.005 # time increment
    P1 = self.dt*self.A*np.exp(-self.k*(action//10)/10) # probabilities of execution
    P2 = self.dt*self.A*np.exp(-self.k*(action%10)/10)
    if random.random() < P1: # decrease inventory increase cash
        self.q -= 1
        self.x += self.s + (action//10)/10
    if random.random() < P2: # increase inventory decrease cash
        self.q += 1
        self.x -= self.s - (action%10)/10
    if random.random() < 0.5:
        self.s += np.sqrt(0.005)*self.sigma
    else:
        self.s -= np.sqrt(0.005)*self.sigma
    self.state = np.array([self.s-100,(self.q-34)/25,(self.t-0.5)/0.29011491975882037])
    reward = self.x+self.q*self.s-(self.oldx+self.oldq*self.olds)
    if np.isclose(self.t, self.T):
        self.done = True
    self.oldq = self.q
    self.oldx = self.x
    self.olds = self.s
    return self.state, reward, self.done, {}

def reset(self):
    self.t = 0.0 # initial time
    self.s = 100.0 # initial midprice value
    self.T = 1.0 # trading period length
    self.sigma = 2 # volatility constant
    self.dt = 0.005 # time step
    self.q = 0.0 # initial inventory
    self.oldq = 0.0
    self.oldx = 0.0
    self.olds = 100.0
    self.x = 0.0 # initial wealth/cash
    self.gamma = 0.1 # risk aversion parameter
    self.k = 1.5 # intensity of arivals of orders
    self.A = 140 # constant
    self.done = False
    self.info = []
    self.state = np.array([self.s-100,(self.q-34)/25,(self.t-0.5)/0.29011491975882037])
    return self.state

The state space is mostly normalized. The action space consists of 100 possible discrete actions (integers from 0 to 99 which are then transformed to (0.0,0.0),(0.0,0.1),...(1.0,1.0). The reward is simply given by the change in the portfolio value (cash+stock).

Note: I've also tried transforming the action space into a continuous one in order to use DDPG but all to no avail.

$\endgroup$
1
  • $\begingroup$ If the environment is too stochastic, then this is the best meaningful result you can get. Else, you may prefer changing the definition of state/action to emulate some particular environment the best in order to hope some luck. $\endgroup$ – RewCie May 6 '20 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.