2
$\begingroup$

The agent is supposed to visit specific locations (which is also different each time) and it may encounters obstacles. The goal is to visit those locations with the shortest path possible without hitting obstacles. Those locations that need to be visited are fruits that requires the agent to fertilize them. There are different fruits but the agent only have to fertilize specific types. The only available information to the agent is its current location and the type of the fruits to be fertilized. It is possible to know the location of the target crops but not the obstacle beforehand. The fixed information in each episode is the size of the field/grid-world, the type and locations of the fruits, the initial state of the agent.

My question is - can I train the agent with a DQN algorithm, avoiding obstacles and still finding the optimal path to finish the task?

Improve this question
$\endgroup$
2
  • $\begingroup$ Could you please give some more information regarding the rules that the obstacles follow? In your previous post you said that the agent can see obstacles, but only nearby ones - is that still the case? Also, it will make a difference to the best approach to use depending one when new locations for obstacles are generated, and what kinds of rule are used to generate those new locations. For example there is a big difference between random positions chosen once at the beginning of each episode, and positions that change on every time stamp. Also between completely random and items that move $\endgroup$
    – Neil Slater
    Commented Jul 3, 2023 at 13:10
  • $\begingroup$ Obstacles are generated at the beginning of each episode and will not each time step. But it does not matter as the agent does not have access to the whole field. It is a simulation which is supposed to be realistic. But for simplicity we assume that the obstacles when they are generated they will not move for the whole episode. About the seeing part, in the training phase at the moment I assume that by giving high penalties, the agent will learn the location of the obstacles and/or not to hit it. But it is not in the state representation of the agent $\endgroup$
    – Mamora
    Commented Jul 3, 2023 at 13:27

1 Answer 1

Reset to default
4
$\begingroup$

This is technically a partially observable MDP (POMDP), because some of the environment state is not visible to the agent. However, there are degrees of non-Markov states, and the environment you describe is pretty close to Markovian, with new details being revealed to the agent effectively at random as time progresses.

So the good news is that a DQN-based agent should be able to cope. One caveat is that you should code the state so that it keeps a memory of the obstacles seen so far (the state is the agent's knowledge of the map and its position on it). Another is that there is no RL solution that will be optimal from the perspective of an observer that knows the location of obstacles in advance. The best any agent can do will be a policy that does well on average given the limited knowledge that the agent has. If a set of obstacles is generated that forms a dead end, the agent may waste time steps discovering that, whilst an observer who can see the obstacles could find a more optimal path that doesn't need to approach the blockage.

There's a difference here between writing an agent to make use of knowledge you as designer has, as opposed to one that learns the hidden state details by itself. It is possible to design agents - DQN or otherwise - that figure out how to represent discoveries in POMDPs by themselves, but it's far easier to code an update to a state/memory yourself if you can and assuming your goal is to solve the environment, not to develop the most sophisticated general learning agent possible.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thank you for your reply. According to your explanation, the best way to proceed is to somehow encode this information related to the obstacles' locations into the code? Otherwise I should go with POMDPS? But then in testing the algorithm how can we add this extra knowledge? What if we make the problem really simple and not looking for the shortest path and instead focusing on fertilizing the target fruits and avoiding the obstacles. I think with my current state representation it cannot solve it. The current state includes the agent location and target fruit type. $\endgroup$
    – Mamora
    Commented Jul 3, 2023 at 14:10
  • 1
    $\begingroup$ @Mamora Make the state the agent's current knowledge of the map, including position of any obstacles it has discovered. If the space is very small, and number of obstacles fixed, you can do this by enumerating all possible states (see the Open AI Taxi environment for something similar - here's a worked solution: towardsdatascience.com/… ). If the environment is larger, e.g. 16x16 grid with 10 or more movable obstacles, then a grid map "image" and CNN for the agent is the way to go. $\endgroup$
    – Neil Slater
    Commented Jul 3, 2023 at 14:24
  • $\begingroup$ The environment is quite big like 300*250 cells and one or two stationary obstacles but being located randomly. $\endgroup$
    – Mamora
    Commented Jul 3, 2023 at 14:36
  • $\begingroup$ So the obstacles are likely to be larger shapes, covering many pixels, and the agent only observe the parts of them that it is adjacent to? 300x250 is a bit large for a starting problem - it may take some effort to tune the CNN and representation. All of this is really different set of questions/answers though - I'd encourage you to try, and perhaps ask again if you get stuck. If I just predict what your challenges are in advance, you miss the chance to try stuff and learn about it even if it doesn't quite work. Plus I could well be wrong $\endgroup$
    – Neil Slater
    Commented Jul 3, 2023 at 18:36
  • $\begingroup$ Thanks @Neil Slater for your answer and the effort. The obstacles are covering several pixels and the idea is that the agent only sees the adjacent parts of it. I am still stuck :D mostly for the state representation. I was thinking about how to proceed, one way is that because the fruits' location and the grid size are known in advance, I can give the robot the initial path to do the task and then whenever it encounters an obstacle it should learn to find a detour to finish the task. But then the state representation should be the whole field? It is too big and I want not to use CNN $\endgroup$
    – Mamora
    Commented Jul 5, 2023 at 8:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .