I am developing AI in the form of NEAT, and it has passed certain tasks like the XOR problem outlined in the NEAT Research Paper. In the XOR Problem, the fitness of a network was determined by an existing function (XOR in this case). It also passed another tests. One I developed was to determine the sine at a certain point X in radians. It also worked, but yet again, its fitness was determined by an existing function (sin (x)).
I've recently been working on training it to play Tic Tac Toe. I decided that to determine its fitness, it would play against a "dumb" AI, placing O's in random locations on the grid, and gaining fitness based on whether or not it placed X's in a valid location (losing fitness if it placed an X on top of another X or an O), and gaining a lot of fitness if it won against the "dumb" AI. This would work, but when a network got really lucky and the "dumb" AI placed O's in impractical locations, the network would win and gain a lot of fitness, making it very difficult for another network to beat that fitness. Therefore, the learning process did not work and I was not able to generate a Tic Tac Toe network that actually worked well.
I do not want the GA to learn based off an "intelligent" tic tac toe AI because the whole point of me training this GA is so that I do not have to make the AI in the first place. I want it to be able to learn rules on its own without me having to hard code an AI to be very good at it.
So, I got to thinking, and I thought it would be interesting if the fitness of a network could be determined based off how well it played against OTHER NETWORKS in its generation. This does seem similar to how humans learn to play games, as I learned to play chess by playing against other people hundreds of times, learning from my mistakes, and my friends also increased in their ability to play chess as well. If GA's were to do that, that would mean I don't have to program AI to play the game (in fact, I wouldn't have to program a "dumb" AI as well, I would only have to hard code the rules of the game, obviously).
My questions are:
Has there been any research or results from GA's determining their fitness based off competing against each other? I did some searching but I have no idea what to look for in the first place (searching 'NEAT fight against each other' did not work well :-( )
Does this method of training a GA seem practical? It seems practical to me, but are there any potential drawbacks to this? Are GA's meant to only calculate predetermined functions that exist, or do they have the potential to learn and do some decision making?
If I were to do this, how would fitness be determined? Say, for the tic tac toe example, should fitness be determined based on whether or not a network places its X's or O's in viable locations, and add fitness if it wins and subtracts fitness if it loses? What about tying the game?
Should networks of the same species compete against each other? If they did, then it would seem impractical to have species in the first place, as networks in the same species competing against each other would not allow a successful species to rise to the top, as it would be fighting against each other.
Kind of out of topic, but with my original idea for the tic tac toe GA, would there be a better way to determine fitness? Would creating an intelligent AI be the best way to train a GA?
Thanks for your time, as this is somewhat lengthy, and for your feedback!