2
$\begingroup$

I'm doing reinforcement learning and have a visual observation as state input for my agent. In the Deepmind Atari paper they greyscale the input image before they input it into the CNN to reduce the input space's size, which makes sense to me.

In my environment i have for each pixel 5 possible states which are represented in black, white, blue, red and green. This also makes intuitive sense to me since it's like a bit-encoding.

Any thoughts on what could be better? greyscaling into 2 shades of grey and black and white also maintains the information, but feels somehow less direct, since my environment's visual space is categorical, which makes more sense in a categorical encoding.

Thank you!

$\endgroup$
  • 1
    $\begingroup$ Greyscaling can only make sense, when the meaning of blue, red and green is somewhere between white and black. Otherwise, you'd force the network to decode it first. And even when the meaning is like above, providing one bit for each possible state sounds better. That's how networks for chess work: one bit per piece kind and field. $\endgroup$ – maaartinus Feb 21 at 5:36
2
$\begingroup$

We can't say for sure which approach would work best in the general case. If you have domain knowledge, you can make a better guess. You'll basically want to answer the question: which information is important for learning an optimal policy?

In my environment i have for each pixel 4 possible states which are represented in black, white, blue, red and green.

Generally, if you have an environment like this, I would (without any other information) guess that each of the 4 (or 5? you mentioned 5 colours?) have some meaning that may be relevant for your agent. That's just what I would guess though. In theory, it might be possible that white means one thing (e.g. "empty"), and every other colour means the same other thing (e.g. "not empty"). If you had domain knowledge like that, and knew that it is only important whether or not any given pixel is white, you could of course binarise your input.

But in general, if the colours might be important, I'd recommend including them. If you really only have a few distinct colours like that though, I would not recommend encoding them in some format like RGB where values can range from 0 to 1 or 0 to 255. I would recommend having 4 (or 5?) binary channels in your input:

  1. Binary channel containing 1s for pixels that are black, and 0s for all other pixels.
  2. Binary channel containing 1s for pixels that are white, and 0s for all other pixels.
  3. Binary channel containing 1s for pixels that are blue, and 0s for all other pixels.
  4. ...
  5. etc.

The reason for this is that deep neural networks often tend to have an easier time learning with binary inputs, and here you can completely binarise your inputs without requiring an excessively high number of channels. If you had hundreds or thousands of different possible colours, this would probably no longer be a good idea.

| improve this answer | |
$\endgroup$
  • $\begingroup$ That's helpful. Indeed should be 5 states. I've edited the question $\endgroup$ – SumakuTension Feb 20 at 9:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.