I have a genetic algorithm that maximizes a fitness function with two variables f(X,Y).
I have been running the algorithm with various parameters in mutation and crossover probability (0.1, 0.2, ...)
Since I dont have much theoretical knowledge of GA, how could I proceed in order to find the optimal values for mutation and crossover probability, and if necessary the optimal population size ?
As @Oliver Mason says, picking the parameters that control the behavior of a GA (which are sometimes called "hyperparameters") is historically more of an art than a science.
The evolutionary computation literature has many theories about the merits of high vs. low mutation, and high vs. low crossover. Most practitioners I have worked with use either high crossover, low mutation (e.g. Xover = 80%, mutation = 5%), or moderate crossover, moderate mutation (e.g. Xover = 40%, mutation = 40%).
In more recent years, the field of hyperparameter optimization has emerged and focuses on developing automatic approaches to picking these parameters. A very simple example of hyperparameter optimization is the GridSearchCV function in ScikitLearn. This systematically tries every combination of, say, 10 crossover values with evey one of 10 mutation values, and reports on which one works best. It uses Cross Validation to prevent overfitting during this process. A more complex approach is Bayesian Hyperparameter Optimization, which performs a sort of optimal experiment design to uncover the best values using as few tests as possible. This approach has been quite successful in tuning the hyperparamters of deep neural networks, for example.
