2
$\begingroup$

I'm trying to train a Policy Gradient Agent with Baseline for my RL research. I'm using the in-built RL toolbox from MATLAB (https://www.mathworks.com/help/reinforcement-learning/ug/pg-agents.html) and have created my own Environment. The goal is to train the system to sample an underlying time-series ($x$) given battery constrains ($\epsilon$ is battery cost).

The general setup is as follows:

  • My Environment is a "sensor" system with exogenous input time-series and battery level as my States/Observations (size is 13x1).
  • Actions $A_t$ are binary: 0 = keep a model prediction $(\hat x)$; 1 = sampling time series $(x)$
  • Reward function is

$$ R = -[err(\tilde x, x) + A_t\cdot \epsilon ] + (-100)\cdot T_1 + (100) \cdot T_2 $$

where $err(\tilde x, x)$ is the RMSE error between the sampled time series $(\tilde x)$, and true time series x.

  • The Terminal State Rewards are -100 if sensor runs out of battery $T_1$ or 100 if reached the end of the episode with RMSE < threshold and remaining battery level $(T_2)$. The goal is to always end in $T_2$.

  • Each training Episode consists of a time-series of random length, and random initial battery level.

My current setup is using mostly default RL setups from MALTAB with learning rate of $10^{-4}$ and ADAM optimizer. The training is slow, and shows a lot of Reward oscillation between the two terminal states. MATLAB RL toolbox also outputs a $Q_0$ value which the state is:

Episode Q0 is the estimate of the discounted long-term reward at the start of each episode, given the initial observation of the environment. As training progresses, Episode Q0 should approach the true discounted long-term reward if the critic is well-designed,

enter image description here

Questions

  • Is my training and episodes too random? i.e., time-series of different lengths and random initial sensor setup.
  • Should I simplify my reward function to be just $T_2$?
  • Why doesn't $Q_0$ change at all?
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.