I am trying to solve the problem of an agent dynamically discovering(start with no information about the environment) the environment and to explore as much of the environment as possible without crashing into obstacles I have the following environment:
where the environment is a matrix. In this the obstacles are represented by 0's and the free spaces are represented with 1s. The position of the agent is given by a label such as 0.8 in the matrix.
Every time it explores the environment it keeps updating its own map:
The single state representation is just the matrix containing-
- 0 for obstacles
- 1 for unexplored regions
- 0.8 for position of the agent
- 0.5 for the places it has visited once
- 0.2 for the places it has visited more than once
I want the agent to not hit the obstacles and to go around them.
The agent should also not be stuck in one position and try to finish the exploration as quickly as possible.
This is what I plan to do:
In order to prevent the agent from getting stuck in a single place, I want to punish the agent if it visits a single place multiple times. I want to mark the place the agent has visited once as 0.5 and if it has visited it more than once that place will be labelled 0.2
The reason I am marking a place it has visited only once as 0.5 is because if there is a scenario where in the environment there is only one way to go into a region and one way to come out of that region, I don't want to punish this harshly.
Given this problem, I am thinking of using the following reward system-
- +1 for every time it takes an action that leads to an unexplored region
- -1 for when it takes an action that crashes into an obstacle
- 0 if it visits the place twice(i.e 0.5 scenario)
- -0.75 is it visits a place more than twice
The action space is just-
Am i right in approaching the problem this way? Is Reinforcement Learning the solution for this problem? Is my representation of the state , action, reward system correct? I am thinking that DQN is not the right way to go because the definition of a terminal state is hard in this problem, what method should I use to solve this problem?