I am trying to code a MCTS agent for tic tac toe and i have some theoretical questions regarding MCTS.
1)I am using the UCB1 MCTS $UCB(Si)=average value + 2*sqrt(ln(N)/ni)$ .
Considering the image below that describes a state diagram,i would like to ask if the UCB(s3)(for state S3) is equal to $3/3 + 2*sqrt(ln(6)/3)$ (i used the total number N of iterations of So ) OR it is $3/3 + 2*sqrt(ln(4)/3)$ (here i used the n1 total number of iterations of the parent node of s3,s4.
- I am wondering what's happening if during the expansion phase of the UCB1 MCTS algorithm i find a state that already exists in the tree. For example: Consider again the image (class diagram below) .During expansion of S1 i find again State s2 .What should i do in this case ? Should i treat s2 as i already visited it and copy its information or treat it like a completely new state ? And when i propagate it after the rollout should i update its stats everywhere?
3)I would like to ask specifically for tic-tac-toe what would be ideal number for the evaluation function.For example considering a ucb1 monte carlo UCB(si)$=average value + 2*sqrt(ln(N)/ni)$ what would be typical numbers for a rollout for a win , a lost and a draw for each player 'x','o'? Thank you in advance