# Find the expected reward in an expectimax-based dice rolling game?

I have this question that I'm kinda stuck on.

It's a game scenario in which we set up an expectimax tree. In the game, you have 3 dice with sides 1-4 that you roll at the beginning. Then, depending on the roll, the player can choose one of the dice to reroll or not reroll anything. Points are assigned like so:

• 10 points if there's 2 of a kind
• 15 if there's 3 of a kind
• 7 if there's a series like 1-2-3 or 2-3-4
• Otherwise, or if the sum is higher than the rewards from above, the score = sum of the rolls

For additional context, this is an example expectimax tree I came up with, for the case that the player rolled a 1,2,4 and is considering rerolling or not:

Now lets introduce a new agent -- a robot that's supposed to help the human player. we assume

• the human player choses any action with uniform probability regardless of the initial roll
• there's a robot that, given a configuration of dice and the human's desired action, actually implements the action with probability 1-p and overrides it with a "no reroll" order with probability p>0. It has no effect if the human's decision is already to not reroll.

For that scenario, I came up with this expectimax tree:

Now for the part I'm actually stuck on -- lets define A, B, C, and D as the expected reward of performing actions "reroll die 1", "reroll die 2", "reroll die 3", and "no reroll." How do we find $$R_H$$, the expected reward for the human acting without the robot's help, and $$R_{AH}$$ the expected reward for if the robot helps? We can't use p in the expression, we only have access to A,B,C,D and we're supposed to write it in the form $$X + Y_p$$

*EDIT: I asked again and the question was worded weirdly. They said we should definitely use p. What was meant by not using p is X and Y themselves can't contain p. But Y will be multiplied by p in the final simplified form.

For $$R_{H}$$ I think the answer should be $$\frac{(A + B + C + D}{4}$$ because of uniform distribution over A-D.

I'm supposing that $$R_{AH}$$ would be $$\frac{(A + B + C)(1-p) + D + Dp}{4}$$? Because the robot doesn't override with probability $$1-p$$, but he can only override A-C and does with probability $$p$$.

I think something feels slightly wrong about my answer but I'm not sure what.

• Are you sure you are not allowed to use p? The expected reward R_AH is a function of p. – Cohensius Feb 24 at 7:29
• Hi. Could you please put your main specific question in the title? "Expectimax World Problem — Dice Rolling Game" is not a question, but the title of an article. – nbro Feb 24 at 13:34
• @Cohensius I asked the instructors and they clarified that you can use p. What was meant by not using p is X and Y themselves can't contain p. But Y will be multiplied by p in the final simplified form. It was a slight wording confusion. I added my best guess at a solution. However, I'm not 100% sure, and would appreciate confirmation :) – Manny Feb 24 at 17:06
• @nbro done! sorry – Manny Feb 24 at 17:06

## 1 Answer

If I understand,

• With probability $$p$$ the robot select no re-roll (action $$D$$).
• With probability $$1-p$$ the human uniformly select an action between $$A,B,C,D$$.

Thus the expected reward for if the robot helps is

$$R_{AH}= p\cdot D+(1-p)\frac{A+B+C+D}{4} = \frac{(A+B+C)(1-p)+D+3pD}{4}$$