Can an agent trained with a DQN algorithm in a grid world, avoid obstacles (randomly appearing during the run time) and still find the optimal path to finish a task? The agent is supposed to visit specific locations (which is also different each time) and it may encounters obstacles. The goal is to visit those locations with the shortest path possible.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
Yes, DQN can do this, but if you want optimal behaviour, you will not be able to use a really simple representation of the state, with only the position of the agent tracked. In order to predict expected returns, and take an optimal path, the agent needs to be told somehow where these obstacles are.
In this kind of scenario it is common to use some "image" of the grid that the agent is navigating, with walls, obstacles and the agent changing the value at the grid point they are occupying. Then the agent can use a convolutional neurtal network (CNN) to approximate Q values. You don't have to do this, any representation that tracks all changeable positions between episodes should work. However, the grid view of the environment is reasonably generic, and would cover many possible versions of the environment (e.g. you can change number of obstacles, add moving obstacles and other grid elements, all using a similar representation and starting agent model).
The training session will need to include new random locations on each episode, so that the agent can learn to generalise how it should deal with them. It will take longer for the agent to learn this generalisation, than if it learned a static maze.
answered Jul 3, 2023 at 7:36
-
$\begingroup$ Thanks for your answer. I see what you mean. I can make all information on the grid world accessible by the agent except the obstacles, cause they are really generated in real time. Then maybe I have to look into solution for partially observable RL solutions?! At the moment I will try to train for many episodes so it can see all the possible locations of the obstacle but then the target locations (locations to be visited) also changes in each run. So it becomes way more complicated $\endgroup$– MamoraCommented Jul 3, 2023 at 7:55
-
$\begingroup$ @Mamora if the agent is allowed to see the obstacles then you can put them on a grid. This is a lot like the DQN Atari agent. If the obstacles have to be invisible to the agent and it needs to learn their position in real time by bumping into them, then I might have some suggestions - ask a new question and give more details of your problem $\endgroup$ Commented Jul 3, 2023 at 8:11
-
$\begingroup$ @Mamora if the agent is allowed to see the obstacles then you can put them on a grid. This is a lot like the DQN Atari agent. If the obstacles have to be invisible to the agent and it needs to learn their position in real time by bumping into them, then I might have some suggestions - ask a new question and give more details of your problem $\endgroup$ Commented Jul 3, 2023 at 8:12
-
$\begingroup$ @Niel Slater, the obstacle is only visible when the agent is close to it. It can see one step ahead. Thanks, I will ask a new question then :) $\endgroup$– MamoraCommented Jul 3, 2023 at 8:13
-
$\begingroup$ @Niel Slater this is the new question with more details about the problem: ai.stackexchange.com/q/41112/73668 $\endgroup$– MamoraCommented Jul 3, 2023 at 9:26