3
$\begingroup$

In reinforcement learning (RL), the reward function (RF), which can be denoted as $r(s)$, $r(s, a)$, $r(s, a, s')$, $r(s, s')$ depending on its specific definition, provides the learning signal, which guides the RL algorithm/agent towards the desirable behaviour. As opposed to a loss (or cost) function in supervised learning, an RF determines the goodness (or quality) of an action in a certain state (in the case of $r(s, a)$) with respect to an optimal behavior/policy. The reward function is typically hand-designed, but it can also be learned with inverse RL techniques.

In evolutionary algorithms, a fitness function (FF) is the function we use to evaluate an individual (or chromosome). The fitness of an individual, a number, represents the quality of that individual.

Both the fitness function and the reward function are very similar, so what's the difference between them? Can a fitness function be thought of as a reward function? Can it be thought of as a cost function in SL problems?

$\endgroup$
2
$\begingroup$

In many cases, a fitness function (FF) is indeed similar to a reward function (RF), but, in other cases, it's more similar to a cost function (CF) as used in supervised learning (SL), and I explain below why. The FF, RF, and CF are used to evaluate the individuals, actions, and predictions, respectively, hence they can all be thought of as evaluation functions (although this latter term has a more specific meaning in the context of search algorithms, like A*). Their primary difference is their purpose and how they are used (i.e. for which problems they are used or what they evaluate), and not much what they return, which is typically a number (in all cases).

Reward function

As stated in the question, an RF can be viewed as evaluating different situations, depending on its definition or what we are interested in.

  • $r(s)$ evaluates the goodness of being state $s$
  • $r(s, a)$ evaluates the goodness of taking $a$ in the state $s$
  • $r(s, s')$ evaluates the goodness of being in state $s$, at the current state time step, then, independently of the action $a$ taken in $s$, ending up in state $s'$
  • $r(s')$ similar to $r(s)$
  • $r(s, a, s')$ the quality of being in state $s$, taking action $a$ and ending up in state $s'$.

Although I will not dwell on the details in this answer (and you can find more details here), depending on the problem or the RL algorithms that you want to use to solve the problem, one definition may be more interesting or useful than another.

Fitness function

In evolutionary algorithms (EA), such as genetic algorithms (GA) or genetic programming (GP), the solutions are generally known as individuals, which can be represented in different ways depending on the specific approach.

However, in all cases, we often distinguish between genotypes (aka genomes or chromosomes), which are representations of the actual solutions to a problem, which are more amenable to changes (i.e. to the mutation and crossover operations), and phenotypes, which are the actual solutions to the problem. The reason why we use genotypes is that, often, it's easier to find solutions in a different but related space rather than in the original space of solutions, although some EAs can also just work with phenotypes.

To be more concrete, in traditional genetic algorithms, the genotypes are binary vectors $\{0, 1 \}^N$, which restricts the space of possible solutions, which, in certain cases, it's a good thing, because the original space may be too huge (so it may be harder to converge to a good solution). These binary vectors can then be converted to the actual solutions using some encoding-decoding mechanism that we need to specify depending on the problem. More precisely, maybe each element of a binary vector corresponds to some binary choice in some game where you need to take a sequence of choices, and each time you can either say "yes" or "no".

So, in EAs, an FF evaluates the quality (or fitness) of an individual and, more precisely, either a genotype or a phenotype (depending on the representation of the solutions).

Often, the fitness function is really a cost function (CF), like in supervised learning. For example, if you want to solve a symbolic regression problem using genetic programming (i.e. find an analytical or closed-form expression of a function), then the fitness function is really just a cost function, i.e. an individual would be like a prediction and the label would be generated by the actual true function (provided that we know it). More concretely, let's say that we want to find the analytical expression

$$f(x) = x^2.$$

In this case, we can generate a labeled dataset of input-output pairs by just evaluating $f$ at different inputs, so we may get the following labeled dataset

$$ D = \{(1, 1), (2, 4), (3, 9) \} $$

We can then evaluate an individual, which in the case of GP is a function (or program) $g(x)$, using e.g. the mean squared error (MSE). For example, if we have the individual $g(x) = x^3$, we can compute the mean squared error as follows

$$ \text{MSE}(D, g(x)) = \frac{1}{3} \left( (1 - 1)^2 + (4 - 8)^2 + (9 - 27)^2 \right) $$

So, in this case, the FF is the MSE. Of course, we could have bigger datasets than $D$. In any case, the point is that you can use GP to solve SL problems.

You could also use EAs to solve other types of problems, i.e. not just SL problems, for instance, RL or unsupervised learning problems. In the case of RL problems, the FF is more similar to an RF in RL, given that you will probably need to define the FF as a function that evaluates the qualities of actions, states, or combinations of these.

So, to conclude, the definition of the FF in EAs depends on the problem that you want to solve, and it can be more similar to an RF or CF, depending on the problem you want to solve.

Further reading

If you are interested in how to design reward functions or fitness functions for your problem, you can take a look at this (and this) and this questions, respectively. This answer provides a brief explanation of the difference between RL and EAs.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.