Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
Sometimes the agent learns a bit slow and you want to have multiple agents in one generation. And at each episode you'll draw on the screen only the best of them or all of them. How is that possible?
For clarification purposes, please watch this video on youtube at time 4:10.
I need just a theoretical approach, I'll try the coding myself :).
Thanks for any answer! I really do appreciate it! :)
The video you linked is not using reinforcement learning (RL). It is using genetic algorithms (GA).
GA is designed around using multiple agents and picking the best performing to move forward to next generation. With this approach, it is common to want to only view the best performing agents, as the learning mechanism uses the same selection process - the best agent is the output of the algorithm*.
Whilst you can run muliple agents to collect more data easily enough in RL, they typically won't perform better or worse other than by random chance. The best performance is not indicative of the agent's overall performance in the way that GA is, because there is only one agent. It would not be as meaningful to pick out the best ones for display. Instead, after a certain number of training episodes, you should take the agent so far, stop it running exploratory moves (set $\epsilon = 0$ if you are using $\epsilon$-greedy exploration). Then render the behaviour of that agent.
If you want to compare RL versus GA for learning efficiency, then one arguably fair comparison would be to render the agent each time it has trained on a number of episodes in RL equal to the population size used in the GA version. If you are only showing results after every 100 generations of GA, then multiply number of training episodes by 100 to compare using the same amount of data input.
You can also, separately to this concern, run RL with multiple exploring agents at once. If you want to have parallel training with multiple environments running in RL, then you will need a distributed environment. Each instance of the environment would run one agent, and collect training data. You have a rough choice between:
Feeding everything observed into one central experience replay memory and a single training loop routine samples from the whole memory and updates the agent. This should work OK for Q-learning.
Calculate update gradients on each distributed environment, collate them centrally and update the agent on a mean gradient update step before sending out the updated agent to all the distributed systems. This is the approach typically used by A3C and A2C which benefit from having multiple agents running at once.
In both cases, the latest parameters of the agent (the neural network weights) need to be regularly copied out to each environment so that each instance can work as much as possible with a current policy.
Setting up a distributed learning environment for RL is more work than for GA, because you need to move a lot more data between agents to complete learning, whilst for GA you only need to measure fitness. However, you should find that Q learning can be a lot more efficient (in terms of number of simulations required) than a GA-based approach for many control problems.
* As an aside, this can be a weakness of GA if you are running in an environment with random choices or events - the GA can select as it's "best" result at any stage something that is less optimal but that was lucky. On average over many generations this effect should be removed, because the same agent won't be luck every generation, but it does mean in some environments that you will get an over-estimate of performance unless your fitness assessment is very thorough (or perhaps run separately every so many generations, to double-check). Similar concerns occur in RL too, but do not affect which agent you select for assessment or display, since there is only ever one set of agent parameters at any time.
