I’m experimenting with reinforcement learning for a 2D pixel plotting task, and am running into an issue that (I think) has to do with the big action space. It goes like this:
The Agent gets two vector inputs each step. Each describes an (n x n) 2d matrix composed of zeros and ones.
One is the (n x n) target matrix, containing a certain shape of zeros The other is an (n x n) state matrix, containing another shape
Every step, I want my agent to pick an (x, y) coordinate: x (picks one of n) y (picks one of n)
This will turn a zero into one, or one into zero.
every step, if correct, I give a small reward, and it’ll get punished when incorrect.
I’m training the agent (a network with 3 layers with 256 hidden units) with PPO, and curiosity in the loss, and for a 12 x 12 matrix it works quite well, not 100% but okay. (see image). Note that the agent doesn't get enough steps here to fully delete the initial shape when the target shape is empty, that's why it doesn't fully make it. Takes about 800K steps to converge though.
But the agent starts struggling in local minima when I increase beyond 32 x 32.
This one is at 32 x 32:
Is this even scalable to bigger matrices even? I was hoping to go 3D eventually, by reaching 100x100x100 .
I do realize that i have a huge input and action space when working with such a grid. Is something like that even possible with an RL paradigm? I’ve tried increasing the network size, and decreasing learning rate, but I’m not satisfied. Any ideas or alternative approaches to plot pixels like this?
Any input is very much appreciated! Thanks!