There are multiple ways to handle 'illegal' individuals, each one with pros and cons:
Abortive methods: The individuals that violate constraints are eliminated as soon as discovered (i.e. after crossover or mutation) and new individuals are generated in order to keep the population stable. This usually implies a slower creation of new generations, as some individuals are discarded.
Contraceptive methods: The crossover and mutation are written in such a way to make it impossible for a newly generated individual to violate any constraints. This way to act is usually more efficient, as no individuals are discarded, but it might not be possible.
Penalization function: The fitness function gives a huge penalization to the individuals that violate constraints (usually proportionally to the constraints it violates). In this way they usually do not get to reproduce and their genes are eventually lost. You'd go this way if illegal individuals are very rare and there is no possibility for them to take over the full population (causing the algorithm to fail).
By experience, try to act on the crossover algorithm first (contraceptive way), it is the best way to exclude constraints violations. If this is not possible, pick one of the other two methods, depending on how often the illegal individuals are generated (not so often -> penalization, very ofter -> abortive) and on how easy it is to penalize individuals that violate constraints (easy -> penalization, not easy -> abortive).
A contraceptive way to handle your example is writing the crossover as following:
const crossover = (p1: boolean[], p2: boolean[]) =>
{
// simple crossover, getting half genome from each parent
const child: boolean[] = [...p1.slice(0,3), ...p2.slice(3,6)];
// contraceptive part
while (countEnabledGenes(child) > 3) child[getRandomIndex()] = false
return child
}
const countEnabledGenes = (genome: boolean[]) => genome.filter(gene => gene).length;
const getRandomIndex = () => Math.round(Math.random() * 6);
But this is a fairly complex problem and each scenario has its own specifications.
