I'm attempting to implement the actor-critic algorithm on Matlab using Radial Basis Function, Local Linear Regression, and shallow Neural Network for inverted pendulum system. the state space and the action space are continuous.

  • states are the angle x_1 wrapped into [-pi pi] and the angle velocity x_2 in [-8*pi 8*pi]
  • the continuous action u, which is bound between [-3 3].
  • reward function is quadrat rho=x'Q x+u'R u where Q=diag(1,5) and R=0.1
  • the desired point is upright position [0 0]'

some notes will be added

  • the used solver is ode45.
  • the sampling time 0.03.
  • it explores random u every step, with normal distribution zero mean sigma=1

    model of the system (to save place the parameters of the model are not written)

     function dy =pendulum(y,u)

function to calculate RBF: the idea is to define centers and widths for N RBFs which cover the entire state space to approximate the value function and policy separately. The RBF is normalized.

function phi=RBF(x,C,B,N)         % x:state, C: centres, B: width, N: nombre of used RBfs
 for i=1:N                        % loop for to calculate the vector phi
     Phi_i=exp(-1/2*(x-C(i,:)')'*B^(-1)*(x-C(i,:)'));  % gaussian function
     Phi_vec=[Phi_vec;Phi_i];                    % not normalized phi vector
     Phi_sum=Phi_sum+Phi_i; % sum for normalisation

 phi=Phi_vec/Phi_sum; % normalized phi vector

 % after tuning the learning rate for actor and critic alpha_a and alpha_c 

 %  every step the following updates shall be carried out: 

 %% generally

 %  Value function V=Theta_O'*RBF(x,C,B,N)

 %  policy pi= Theta_v'*RBF(x,C,B,N)

 % determine u(k) with exploration term

 %% aplly u(k) and gain x(k+1)

 [t,y] = ode45(@(t,y) pendulum(y,u(k)),tspan,x(k)');
 x(k+1,1)= wrapToPi(x(k+1,1)); % wrpping to pi

 % determine Temporal difference Error 

 % eligibility trace

 % Critic update

 %actor update
 Theta_V=Theta_V+alpha_a*Delta(k)*Delta_U(k- 1)*RBF(x,C,B,N);
  • $\begingroup$ Hi Khaled, welcome to AI Stack Exchange. That's a complex system you have implemented, with a lot of code and places to go wrong. In addition, the target environment could have a large impact on suitability of the solution. As it stands, there is no way to answer your question. You need to break the problem down into smaller parts, double-check how confident you are in each part, and ask questions with far more details about parts where you are less confident. $\endgroup$ Apr 11 '19 at 10:17
  • $\begingroup$ If you want to present an overview to see if there is an obvious place to look in your implementation, then I think you need to share: (1) The nature of the environment, including states, actions and rewards. (2) What fields are in your raw data. (3) How you have applied RBF to that data (especially how you have normalised or scaled it). (4) A learning graph for actor-critic. Use edit to add those details to the current question $\endgroup$ Apr 11 '19 at 10:27
  • $\begingroup$ thank you for considering my post and advice me to improve it $\endgroup$
    – Khaled
    Apr 11 '19 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.