I have an unusual but very interesting problem. I have a game that is very similar to Toon Blast (a puzzle mobile game). It's based on a Match-2 mechanic in which you can destroy 2 or more connected blocks and your goal is to complete all the required objectives (collect X color blocks, destroy 30 balloons, etc).
I have tons of levels and the ML solver seems to perform very well for all kinds of obstacles - except Quicksand.
Quicksand is a special object that replicates itself to a nearest tile whenever a user makes a move. If the user destroyed a quicksand in his turn, a quicksand won't be replicated. So basically the fastest way to destroy quicksand is to make sure you destroy as much quicksand as you can in each turn so it won't replicate and cover your board.
I use ML-Agents from Unity (https://github.com/Unity-Technologies/ml-agents) and I just give the agent reward=1f whenever it completes an objective (destroy 1 obstacle) and I subtract 1f from reward whenever it performs a move.
For simpler non-replicating obstacles it works perfectly. For example, you click 2 blocks next to a balloon - it will pop a balloon, add 1f as a reward and at the same time remove 1f for using a move.
This way the agent learns to make as few moves as possible.
Below you'll find how the Quicksand works and some simple obstacle - Balloon.
Quicksand preview (sorry for bad quality, 2mb max size)
Balloon preview
My issue is that no matter what, I can't teach it to solve quicksand. With the above rewarding approach, I think this strange creature learned that by actually REPLICATING quicksand it's gaining more reward because it can destroy it later (actually it's not because moves give -1f so it's more-less equal).
I've tried not giving reward for the quicksand, so it only loses rewards by using moves. But it doesn't work either, I'm not sure why.
Do you guys have any idea how this kind of things should be taught?
