Use of reinforcement learning makes most sense over LSTM if control parameters can change during brewing to optimize results, which is not standard brewing policy in low end brewers but more precise control of brewing may produce superior results, especially with regard to temperature control.
Policy does not necessarily need to be predisposed to brewing, if the search is guided by a deep artificial network (DNN), with parameters $\theta$ randomly initialized and defined as follows.
$$ (p, v) = f_{\theta}(s) $$
The brewing process parameters comprise state $s$. The vector of control change probabilities $p$ have components $p_a = \Pr(a|s)$ for each action $a$. A scalar $v$ estimates the value of the expected outcome $o$ from control state $s$, where $v \approx \mathbb{E} [o|s].
Rather than characterize the variations in brewing stock as disturbance $D$, it is more consistent with most policy learning approaches to consider the measured properties of brewing stock as features $\vec{b}$ and feed a normalized signal $\mathcal{N}(b)$ into additional inputs to the deep network. This way, sensitivity to variations in $\vec{b}$ is learned as a critical factor in brewing policy.
In such a case, a vanilla MCTS (Monte-Carlo tree search algorithm) can be employed. Using a deep network removes the need to extract domain specific knowledge or discover heuristics to effectively guide control change policy.
Control values must be made discrete, based on granularity that fully (but not excessively) covers brewing possibilities. The control state probabilities that guide the search can be learned using coffee drinker feedback collected in any number of ways, depending the relative costs of each form of outcome data acquisition.
A focus group
A group of independent testers
Repurchase statistics at a selected set of coffee shops
Feedback controls at a set of offices that provide employees with coffee
For the comprehensiveness in control response to variations in brewing stock properties, the distribution of component values in $\vec{b}$ and their statistical correlation over time may need to be controlled during the training process such that trends and distributions resemble likely real world variations in brewing stock.
Each search during the training process is guided by MCTS control changes, $a_t ∼ \pi_t$. The final state at the end of training is $s_T$, rated using a fixed scale of taster feedback. Selection of each state $s$ can be based on a control operation search advantage based on three characteristics.
Low past experience (a low count of previous visits in the search)
High probability $p$
High average of values $\{v, \; ...\}$ over the leaf states of brewing trials that selected $a$ from $s$ according to the converging neural network parameters $f_{\theta}$
The output of the search is $\vec{\pi}$ representing a probability distribution over control changes. The deep network parameters $\theta$ converge via gradient descent using a loss function $\ell$.
$$ \ell = {(o − v)}^2 − {\pi}_T \, \ln{p} + c \, {||\theta||}^2 $$
Loss is thus an aggregation of two optimality criteria.
Mean square estimated disparity between the predicted outcome $v_t$t and the taste outcome $o$
Cross-entropy of policy vector $p_t$ and search probabilities ${\pi}_t$
Hyper-parameter $c$ weighted L2 regularization
The updated parameters are used in subsequent brew and taste trials. All control operations must be included in each trial. Success using this approach has been obtained using either greedy or proportionality with respect to the root visit counts.
