I'm attempting to implement the actor-critic algorithm on Matlab using Radial Basis Function, Local Linear Regression, and shallow Neural Network for inverted pendulum system. the state space and the action space are continuous.
- states are the angle x_1 wrapped into [-pi pi] and the angle velocity x_2 in [-8*pi 8*pi]
- the continuous action u, which is bound between [-3 3].
- reward function is quadrat rho=x'Q x+u'R u where Q=diag(1,5) and R=0.1
- the desired point is upright position [0 0]'
some notes will be added
- the used solver is ode45.
- the sampling time 0.03.
it explores random u every step, with normal distribution zero mean sigma=1
model of the system (to save place the parameters of the model are not written)
function dy =pendulum(y,u) dy(1,1)=y(2); dy(2,1)=1/J*(M*g*l*sin(y(1))-(b+K^2/R)*y(2)+K/R*u); end_function
function to calculate RBF: the idea is to define centers and widths for N RBFs which cover the entire state space to approximate the value function and policy separately. The RBF is normalized.
function phi=RBF(x,C,B,N) % x:state, C: centres, B: width, N: nombre of used RBfs Phi_vec=; Phi_sum=0; for i=1:N % loop for to calculate the vector phi Phi_i=exp(-1/2*(x-C(i,:)')'*B^(-1)*(x-C(i,:)')); % gaussian function Phi_vec=[Phi_vec;Phi_i]; % not normalized phi vector Phi_sum=Phi_sum+Phi_i; % sum for normalisation end phi=Phi_vec/Phi_sum; % normalized phi vector
% after tuning the learning rate for actor and critic alpha_a and alpha_c % every step the following updates shall be carried out: %% generally % Value function V=Theta_O'*RBF(x,C,B,N) % policy pi= Theta_v'*RBF(x,C,B,N) % determine u(k) with exploration term u(k)=Theta_V'*RBF(x,C,B,N)+Delta_U(k-1) %% aplly u(k) and gain x(k+1) [t,y] = ode45(@(t,y) pendulum(y,u(k)),tspan,x(k)'); : x(k+1,1)= wrapToPi(x(k+1,1)); % wrpping to pi % determine Temporal difference Error Delta(k)=r(k)+gamma*Theta_O'*RBF(x(k),C,B,N)-Theta_O'*RBF(x(k-1),C,B,N); % eligibility trace z=lamda*gamma*z+RBF(x,C,B,N); % Critic update Theta_O=Theta_O+alpha_c*Delta(k)*z; %actor update Theta_V=Theta_V+alpha_a*Delta(k)*Delta_U(k- 1)*RBF(x,C,B,N);