# Understanding policies in helicopter control in the paper by Andrew Ng et al

I was going through this paper on helicopter flight control using reinforcement learning by Andrew Ng et al.

It defines two policy classes to learn two policies, one for hovering the helicopter and another for maneuvering (aka trajectory following). The goal for hovering policy is defined as follows:

We want a controller that, given the current helicopter state and a desired hovering position and orientation $$\{x^*, y^*, z^*, \omega^*\}$$ computes controls $$a\in [-1,1]^4$$ to make it hover stably.

The goal for maneuvering policy is given in term of hovering policy as follows:

Given a controller for keeping a system’s state at a point $$(x^*,y^*,z^*,\omega^*)$$, one standard way to make the system move through a particular trajectory is to slowly vary $$(x^*,y^*,z^*,\omega^*)$$ along a sequence of set points on that trajectory points.

The neural network for these policy classes is shown as follows:

In this, $$(\dot{x},\dot{y},\dot{z})$$ are velocity estimates, $$(\phi,\theta)$$ is helicopter roll and pitch estimates and $$\dot{w}$$ is angular velocity component estimate (more on this at the bottom of page 1 of the linked paper).

Each edge with an arrow in the picture denotes a tunable parameter. The solid lines show the hovering policy class. The dashed lines show the extra weights added for trajectory following (maneuvering). With this observation, I had following doubts:

Q1. Does addition of dashed lines to hovering policy to get maneuvering policy makes manuvering policy superset of hovering policy?

Q2. Rephrasing Q1: can we use maneuvering policy for hovering task (say by setting weights corresponding to dashed lines to zero)?

Q3. If maneuvering policy is indeed a superset of hovering policy, why the authors dont use just maneuvering policy for both tasks or maneuvering policy for hovering task also? Is it because it involves computation involving helicopter's additional sub dynamics represented by dashed line and this additional computation is not required for hovering task?

Or am I completely getting wrong with all these questions?

• Too many questions in the same post. Please, edit this post to leave only 1 question. Ask the other questions in their separate post, even though they are all related. – nbro Feb 10 at 23:33
• All questions are very tightly related. So I felt asking them together will help answerer to stay in the larger context and answer more holistically. Asking them as separate post will require me to copy paste everything (the related context from the paper) before Q1 in three different posts. Is such duplication preferred on stackexchange? – Rnj Feb 11 at 8:40
• It's fine to provide enough context to be able to answer your question in each post, even if that means somehow that there will be a bit of duplication. In my opinion, what's not fine is that you ask us to put the effort to answer multiple questions. Unless someone has read or reads the full paper, which requires some time and effort, it's difficult to answer any of the questions at all. So, that's why I'm suggesting you to simplify this post as much as possible and ask only 1 main specific question. Moreover, I would suggest that you put your main specific question in the title. – nbro Feb 11 at 12:22
• As a rule of thumb, if you can't do that, then it means that your post is too broad. Right now, "Understanding policies in helicopter control in the paper by Andrew Ng et al" is like the title of a blog post where you explain everything of one paper. – nbro Feb 11 at 12:22