0
$\begingroup$

Pseudo code for Cross entropy method according to youtube lecture 32:55

  1. Initialize $\mu \in R^{d}, \sigma \in R^{d}$
  2. iteration 1,2,...
  3. Collect n samples of $\theta_{i} \sim N(\mu,diag(\sigma))$
  4. Perform Noisy evaluation $R_{i} \sim \theta_{i}$
  5. Select the top p% of samples (e.g. p = 20), which we'll call the elite set
  6. Fit a Gaussian distribution, with diagonal covariance, to the elite set, obtaining a new $\mu, \sigma$
  7. Return the final $\mu$

Here is what I am doing:

  1. Create a neural network that takes in the state and gives out the action
  2. Create separate vectors for mean and std of all the network parameters. (i.e. a 1-d vector that contains mean and std for each parameter of neural net)
  3. Repeat n times:
    1. Initialize network parameters by sampling from gaussian distribution corresponding to all the parameters (using their mean and std defined in step 2) (let this be $\theta_{i}$)
    2. Get a single trajectory from this neural network, and use this as the cost of the $\theta_{i}$ (some number of timesteps)
  4. Select top p% trajectories as the elite set
  5. Take mean and standard deviation of their parameter vectors.
  6. Save this new mean and standard deviation vector to be used in step 2 next time. (when finding action for another state)

Questions:

  1. Is this a valid cross entropy method algorithm if this is run at every query for getting action?
  2. Is taking just a single trajectory for each parameter enough in step 3.2?
  3. Given that the parameters come from a gaussian distribution all the time, is there an analytic solution to the problem of getting the best parameters or sampling is required? (Section 4 of this paper Cross Entropy Method for fast policy search
  4. Can step 3 be parallelized?
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.