This is more of a general question of how to model/preprocess 'visual' state-observations to an Agent in Reinforcement Learning that I'll illustrate with an example.

Say you have a reinforcement learning problem where the agent has to draw pixels in an n * n 2D state-matrix of 0's and 1's. Say n = 100. The agent can move one step (up, down, left, right) and on its location can additionally switch 0's into 1's or the other way around.

Each step, it needs to take action so that the state-matrix resembles an n * n target-matrix (that has a certain shape). It is rewarded accordingly each step.

The agent will know its location from an x and y position that are given in addition to the state- and target-matrix each step.

Now I'm curious to the question what the best way is to represent the state to the agent. Using a visual 'prior', or not. Here's two ways:

  1. Based on that you want to give only the essential information to the agent: The agent is presented with a matrix (with target subtracted from state), that will be flattened into one array of n^2. Additionally it'll know its current location as an additional (x, y) vector observation.

  2. Based on that (1) would be more difficult to solve for a human, because you'll have to learn from a flattened array how different points are connected (think about how hard a flattened game of chess would be), you can also use a convolutional neural network to encode the current scene. In this case the agent will be e.g. a red dot. Given that it's such a visual task, it seems to me that using this would give the agent a better model of how the environment works, since the spatial relations are kept intact. Also it feels that keeping the 2D shape intact with a CNN would mean that it'd form better representations that generalize to other shapes, but I can't really say why.

On the other hand one could say that it's arrogant to assume that our 'human' spatial way of interpreting visual information is the best way for this case. Maybe there's a mathematical solution?

Any ideas?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.