# How to add more than 1 agent in one generation with Q Learning

Sometimes the agent learns a bit slow and you want to have multiple agents in one generation. And at each episode you'll draw on the screen only the best of them or all of them. How is that possible?

For clarification purposes, please watch this video on youtube at time 4:10.

I need just a theoretical approach, I'll try the coding myself :).

Thanks for any answer! I really do appreciate it! :)

The video you linked is not using reinforcement learning (RL). It is using genetic algorithms (GA).

GA is designed around using multiple agents and picking the best performing to move forward to next generation. With this approach, it is common to want to only view the best performing agents, as the learning mechanism uses the same selection process - the best agent is the output of the algorithm*.

Whilst you can run muliple agents to collect more data easily enough in RL, they typically won't perform better or worse other than by random chance. The best performance is not indicative of the agent's overall performance in the way that GA is, because there is only one agent. It would not be as meaningful to pick out the best ones for display. Instead, after a certain number of training episodes, you should take the agent so far, stop it running exploratory moves (set $$\epsilon = 0$$ if you are using $$\epsilon$$-greedy exploration). Then render the behaviour of that agent.

If you want to compare RL versus GA for learning efficiency, then one arguably fair comparison would be to render the agent each time it has trained on a number of episodes in RL equal to the population size used in the GA version. If you are only showing results after every 100 generations of GA, then multiply number of training episodes by 100 to compare using the same amount of data input.

You can also, separately to this concern, run RL with multiple exploring agents at once. If you want to have parallel training with multiple environments running in RL, then you will need a distributed environment. Each instance of the environment would run one agent, and collect training data. You have a rough choice between:

• Feeding everything observed into one central experience replay memory and a single training loop routine samples from the whole memory and updates the agent. This should work OK for Q-learning.

• Calculate update gradients on each distributed environment, collate them centrally and update the agent on a mean gradient update step before sending out the updated agent to all the distributed systems. This is the approach typically used by A3C and A2C which benefit from having multiple agents running at once.

In both cases, the latest parameters of the agent (the neural network weights) need to be regularly copied out to each environment so that each instance can work as much as possible with a current policy.

Setting up a distributed learning environment for RL is more work than for GA, because you need to move a lot more data between agents to complete learning, whilst for GA you only need to measure fitness. However, you should find that Q learning can be a lot more efficient (in terms of number of simulations required) than a GA-based approach for many control problems.

* As an aside, this can be a weakness of GA if you are running in an environment with random choices or events - the GA can select as it's "best" result at any stage something that is less optimal but that was lucky. On average over many generations this effect should be removed, because the same agent won't be luck every generation, but it does mean in some environments that you will get an over-estimate of performance unless your fitness assessment is very thorough (or perhaps run separately every so many generations, to double-check). Similar concerns occur in RL too, but do not affect which agent you select for assessment or display, since there is only ever one set of agent parameters at any time.

• I am not sure whether RL and GA can be compared. As far as my understanding of GA goes the agents are traversing over a well defined value function to search for lowest values or highest, whereas in RL you are actively finding out values which are very sketchy/noisy and then creating a policy to maximize on those sketchy values. Correct me if I am wrong. – DuttaA May 20 at 16:39
• @DuttaA: If your GA finds a policy in an environment, then you can definitley compare the policies for performance. It doesn't matter how you produced a policy at all. Deeper comparisons are harder, which is why it is only a suggestion that a fair-ish approach is to compare policies after considering the same amount of data. Other comparisons might be total CPU (where GAs do better because the update steps are less expensive) or development time etc. – Neil Slater May 20 at 16:42
• I guess I'll have to read more about GAs before replying, but as far as I remember GAs have no concept of policy. – DuttaA May 20 at 16:47
• @DuttaA: It is also possible to apply GAs to optimisation problems that are not really suitable for RL, but in this answer I am assuming that there is a control problem suitable for both. You are right GA does not have anything to say about policy functions directly. But provided you can model the function in a way that can be evolved (see NEAT) then a GA can be used to search for optimal results of that function – Neil Slater May 20 at 16:47
• @DuttaA As Neil says, in a similar fashion to algorithms such as simulated annealing, GAs are really a very general approach for solving optimization problems. You just need some way of comparing one solution to another (e.g. you don't need gradient-based information). – nbro May 20 at 16:55