Reinforcement Learning (RL) at its core does not have anything directly to say about adversarial environments, such as board games. That means in a purely RL set up, it is not really possible to talk about the "strength" of a player.
Instead, RL is about solving consistent environments, and that consistency requirement extends to any opponents or adversarial components. Note that consistency is not the same as determinism - RL theory copes well with opponents that effectively make random decisions, provided the distribution of those decisions does not change based on something the RL agent cannot know.
Provided an opponent plays consistently, RL can learn to optimise against that opponent. This does not directly relate to the "strength" of an opponent, although usually strong opponents present a more challenging environment to learn overall.
- Wouldn't the agent (after learning) be able to play only with that opponent and win? since it estimated the value function of this specific behavior only.
If the RL has enough practice and time to optimise against the opponent, then yes the value function (and any policy based on it) would be specific to that opponent. Assuming, the opponent did not play flawlessly, then the RL would learn to play such it would win as often as possible against the opponent.
When playing against other opponents, the success of the RL agent will depend on how similar the new opponent was to the original that it trained against.
- would it be able to play as good with weaker players?
As stated above, there is not really a concept of "stronger" or "weaker" in RL. It depends on the game, and how general the knowledge is that strong players require in order to win.
In theory you could construct a game, or deliberately play strongly, but with certain flaws, so that RL would play very much to counter one play style, and would fail against another player that did not have the same flaws.
It is difficult to measure this effect, because human players learn from their mistakes too, and are unlikely to repeat the exact same game time after time, but with small variations at key stages. Humans do not make consistent enough opponents, and individual humans do not play enough games at each stage of their ability to study fine-grained statistics of their effective policies.
In practice it seems likely that the effect of weakening against new players would be there in RL, due to sampling error if nothing else. However, it seems that the "strength" of players as we measure them in any game of skill such as chess or go, does correlate with a generalised ability. In part this is backed up by consistent results with human players and Elo ratings.
Any game where you can form "rings" of winning players:
- Player B consistently beats Player A
- Player C consistently beats Player B
- Player A consistently beats Player C
Could cause issues of the type you are concerned about when applying RL to optimise an artificial agent.
- How do you develop an agent that can estimate a value function that generalizes against any behavior and win?
If is possible to play perfectly, then a value function which estimated for perfect play would work. No player could beat it. Think of Tic Tac Toe - it is relatively easy to construct perfect play value functions for it.
This is not achievable in practice in more complex games. To address this, and improve the quality of its decisions, what AlphaGo does is common to many game-playing systems, using RL or not. It performs a look-ahead search of positions. The value function is used to guide this. The end result of the search is essentially a more accurate value function, but only for the current set of choices - the search focuses lots of computation on a tiny subset of all possible game states.
One important detail here is that this focus applies at run time whilst playing against any new opponent. This does not 100% address your concerns about differing opponents (it could still miss a future move by a different enough opponent when searching). But it does help mitigate smaller statistical differences between different opponents.
This search tree is such a powerful technique that for many successful game playing algorithms, it is possible to start with an inaccurate value function, or expert heuristics instead, which are fixed and general against all players equally. IBM's Deep Blue is an example of using heuristics.
self-play? if yes, how does that work?
Self-play appears to help. Especially in games which have theoretical optimal play, value functions will progress towards assessing this optimal policy, forming better estimates of state value with enough training. This can give a better starting point than expert heuristics when searching.