4
$\begingroup$

Summary: I am teaching bots to pick food on a playing field. Some food is poisonous and some is good.

Food Details: Poisonous food subtracts score points and good food adds. Food points vary based on it's size. There is about 9:1 ratio of poisonous food to good food, so a lot more chances to end up in negative numbers. Food grows in points overtime. Food spoils after some predetermined size becoming poisonous.

Fitness Function: The fitness function I use is simply counting points by the end of iterations. Bot's might choose to eat it or skip it.

The Problem: The problem I am having is that first generation, most bots eat a lot of bad crap and the curious ones end up in negative numbers. So mostly the ones that make it are the ones that are lazy and didn't eat or didn't head towards the food and most of the time the fittest for first few generations comes out with 0 points and 0 eats of any kind of food. When trained for long time they just end up waiting for the food instead of eating multiple times. Often while they wait food goes bad and they just end up going to another food. This way in the end of the iteration, I have some winners but they are nowhere near the potential they could have been at.

Question: I somehow need to weight the importance of eating food. I want them to eventually learn to eat.

So I thought of this:

brain.score += foodValue * numTimesTheyAteSoFar

But this blows up the score too much and now the food quality is not respected and they just gulp on anything slightly above 0.

Please help.

$\endgroup$
  • 1
    $\begingroup$ Welcome to Stack:AI! (I literally just created that NEAT tag--glad it's finding use :) $\endgroup$ – DukeZhou Mar 5 '18 at 20:22
  • 1
    $\begingroup$ Ah, I was thinking why was there just one or two questions - almost though this StackExchange is not that populated yet :) Thanks for good timing :) $\endgroup$ – Alexus Mar 5 '18 at 21:24
4
$\begingroup$

A human analogy can help you here (a variance).
Initialize all the agents with an initial value x, we will call this energyUnits. Will talk later more on this. Now add some value, as an incentive, whenever the agent eats a good food, to the energyUnits. You need to add a function that will keep decrementing the value of the agent's energyUnits, as human degrade energy (calories) with time. We will call this function normalDegrade. This is the core part of the solution for your problem. Now, for the bad (or poisonous) food you can be more creative with. You can simply subtract a given value whenever an agent eats poisonous food. Or you can extend your normalDegrade function with a very high downward slope. In this case, the energy units (value) of the agent will fall very rapidly. This will force the agent to look for good food to survive.
Since the ratio of food is 9:1 with poisonous in advantage you need to initialize the value of x (energyUnits) very high. You need to do some trial and error to find the right fit for you here. Also, I am assuming that the agent is being removed from the population whenever the value of x is zero or some negative value (which depends). This is important as it makes sure that the algorithm is not wasting time in processing bad agents.
Because of this, another problem arises of the population coming to extinct. For this, you need to keep generating new agents for which any of the Genetic Algorithm will do. New population with better parents of the already present generation will keep the population fit and efficient.

A good fitness function is a core to solving any problem of this kind and sometimes it is hard to find. You might need to do some trial and error with different values to look for the right fit.

$\endgroup$
  • 2
    $\begingroup$ Thank you! This solved it. I shortened the time frame at which the evaluation happened thus forcing the lazy ones out on early stages. Time function to decrease the value helped a lot! $\endgroup$ – Alexus Mar 6 '18 at 21:33
0
$\begingroup$

If i understand your problem right, you have a fitness function for a food-collecting game and wants to learn the weights for the scoring function. This has to be done against something. The baseline for playing a game right, is a human demonstration. The first step is, to build into the game a manual mode, in which the bots are controlled by a human. The next step is to record the demonstrations in an episodic memory. Clustering the overall memory results into the needed weights.

But let us go into the details. The problem in the OP is a pacman like game, with more bots at the same time. For storing the game-logs into a database a simple 3x3 pixel array is well suited. After recording some human-demonstration, the episodic memory contains patterns which are showing, how the player reacts in a certain situation. For example, he collects only the healthy food. The similarity between the current game situation and the stored previous game-logs are fine-tuned with weights. They can be adjusted manual or automatic. Enhancing intelligent agents with episodic memory, page 40 and page 49

$\endgroup$
  • 1
    $\begingroup$ That's not quite what I am looking for. I have a generation of bots that eat food. I am just looking to eventually using NEAT algorithm train a single smartest bot that will efficiently collect good food and omit bad food while not skipping good food in hopes that it will pick up a larger chunk further down the road. $\endgroup$ – Alexus Mar 6 '18 at 21:32
  • 1
    $\begingroup$ Trust me, you're are looking for an episodic memory (aka case-based reasoning). Because the NEAT algorithm needs a cost-function, otherwise he can't convergate to a policy. And trying out all possible neural networks with a 2 Ghz PC isn't possible. Finding a cost function for NEAT is called in literature “inverse reinforcement learning”, and is done with expert-demonstrations. $\endgroup$ – Manuel Rodriguez Mar 7 '18 at 7:27
  • $\begingroup$ So you mean that there is no way to adjust the fitness function to just "breed" the network that does ok with this task? $\endgroup$ – Alexus Mar 7 '18 at 22:06
  • 1
    $\begingroup$ @Alexus: Adjusting the fitness function is exactly the goal which results into a working controller. A possible implementation uses domain knowledge for guiding the neuroevolution process. (1) page 5 The main problem with a vanilla fitness function is the delayed reward. That means the learning controller get only unqualified feedback. $\endgroup$ – Manuel Rodriguez Mar 8 '18 at 7:51
  • $\begingroup$ At first I though you sir don't understand my question, but now I get it that I didn't understand your answer - you are genius and seem to be way above what I know about AI - going to rad that entire paper - thank you! $\endgroup$ – Alexus Mar 8 '18 at 21:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.