# Model Based rl and cross entropy method with nonlinear function approximators

Pseudo code for Cross entropy method according to youtube lecture 32:55

1. Initialize $$\mu \in R^{d}, \sigma \in R^{d}$$
2. iteration 1,2,...
3. Collect n samples of $$\theta_{i} \sim N(\mu,diag(\sigma))$$
4. Perform Noisy evaluation $$R_{i} \sim \theta_{i}$$
5. Select the top p% of samples (e.g. p = 20), which we'll call the elite set
6. Fit a Gaussian distribution, with diagonal covariance, to the elite set, obtaining a new $$\mu, \sigma$$
7. Return the final $$\mu$$

Here is what I am doing:

1. Create a neural network that takes in the state and gives out the action
2. Create separate vectors for mean and std of all the network parameters. (i.e. a 1-d vector that contains mean and std for each parameter of neural net)
3. Repeat n times:
1. Initialize network parameters by sampling from gaussian distribution corresponding to all the parameters (using their mean and std defined in step 2) (let this be $$\theta_{i}$$)
2. Get a single trajectory from this neural network, and use this as the cost of the $$\theta_{i}$$ (some number of timesteps)
4. Select top p% trajectories as the elite set
5. Take mean and standard deviation of their parameter vectors.
6. Save this new mean and standard deviation vector to be used in step 2 next time. (when finding action for another state)

Questions:

1. Is this a valid cross entropy method algorithm if this is run at every query for getting action?
2. Is taking just a single trajectory for each parameter enough in step 3.2?
3. Given that the parameters come from a gaussian distribution all the time, is there an analytic solution to the problem of getting the best parameters or sampling is required? (Section 4 of this paper Cross Entropy Method for fast policy search
4. Can step 3 be parallelized?