Actor-critic algorithm using gaussian Radial Basis Function, Local Linear Regression and shallow Neural Network

I'm attempting to implement the actor-critic algorithm on Matlab using Radial Basis Function, Local Linear Regression, and shallow Neural Network for inverted pendulum system. the state space and the action space are continuous.

• states are the angle x_1 wrapped into [-pi pi] and the angle velocity x_2 in [-8*pi 8*pi]
• the continuous action u, which is bound between [-3 3].
• reward function is quadrat rho=x'Q x+u'R u where Q=diag(1,5) and R=0.1
• the desired point is upright position [0 0]'

some notes will be added

• the used solver is ode45.
• the sampling time 0.03.
• it explores random u every step, with normal distribution zero mean sigma=1

model of the system (to save place the parameters of the model are not written)

function dy =pendulum(y,u)
dy(1,1)=y(2);
dy(2,1)=1/J*(M*g*l*sin(y(1))-(b+K^2/R)*y(2)+K/R*u);
end_function

function to calculate RBF: the idea is to define centers and widths for N RBFs which cover the entire state space to approximate the value function and policy separately. The RBF is normalized.

function phi=RBF(x,C,B,N)         % x:state, C: centres, B: width, N: nombre of used RBfs
Phi_vec=[];
Phi_sum=0;
for i=1:N                        % loop for to calculate the vector phi
Phi_i=exp(-1/2*(x-C(i,:)')'*B^(-1)*(x-C(i,:)'));  % gaussian function
Phi_vec=[Phi_vec;Phi_i];                    % not normalized phi vector
Phi_sum=Phi_sum+Phi_i; % sum for normalisation

end
phi=Phi_vec/Phi_sum; % normalized phi vector

% after tuning the learning rate for actor and critic alpha_a and alpha_c

%  every step the following updates shall be carried out:

%% generally

%  Value function V=Theta_O'*RBF(x,C,B,N)

%  policy pi= Theta_v'*RBF(x,C,B,N)

% determine u(k) with exploration term
u(k)=Theta_V'*RBF(x,C,B,N)+Delta_U(k-1)

%% aplly u(k) and gain x(k+1)

[t,y] = ode45(@(t,y) pendulum(y,u(k)),tspan,x(k)');
:
x(k+1,1)= wrapToPi(x(k+1,1)); % wrpping to pi

% determine Temporal difference Error
Delta(k)=r(k)+gamma*Theta_O'*RBF(x(k),C,B,N)-Theta_O'*RBF(x(k-1),C,B,N);

% eligibility trace
z=lamda*gamma*z+RBF(x,C,B,N);

% Critic update
Theta_O=Theta_O+alpha_c*Delta(k)*z;

%actor update
Theta_V=Theta_V+alpha_a*Delta(k)*Delta_U(k- 1)*RBF(x,C,B,N);
• Hi Khaled, welcome to AI Stack Exchange. That's a complex system you have implemented, with a lot of code and places to go wrong. In addition, the target environment could have a large impact on suitability of the solution. As it stands, there is no way to answer your question. You need to break the problem down into smaller parts, double-check how confident you are in each part, and ask questions with far more details about parts where you are less confident. – Neil Slater Apr 11 at 10:17
• If you want to present an overview to see if there is an obvious place to look in your implementation, then I think you need to share: (1) The nature of the environment, including states, actions and rewards. (2) What fields are in your raw data. (3) How you have applied RBF to that data (especially how you have normalised or scaled it). (4) A learning graph for actor-critic. Use edit to add those details to the current question – Neil Slater Apr 11 at 10:27
• thank you for considering my post and advice me to improve it – Khaled Apr 11 at 15:31