# How does reinforcement learning handle measured disturbances?

I recently encountered an interesting problem and was wondering how RL would solve it. The objective of the problem is to maximize the coffee quality, given by box X. The coffee quality objective function is defined by the company.

To maximize the quality of the coffee, we can perform 2 actions:

• change stirring speed of the coffee machine
• change the temperature of the coffee machine

Now to the tricky part. The coffee bean characteristic coming into the coffee machine is random. We can measure its characteristics before sending them to the coffee machine, but we cannot change them.

I formulated this problem into a control problem, such that $$X$$ is a function of my previous states, $$X(t - k)$$, the control input , $$U$$, and the measured disturbance, $$D$$. Given a constant $$D$$, the problem is trival to solve because the disturbance is a constant part of the environment. However, during times when $$D$$ changes rapidly, the policy is no longer optimal.

How do I inject the information of the measured disturbance into my RL agent? ## 1 Answer

Use of reinforcement learning makes most sense over LSTM if control parameters can change during brewing to optimize results, which is not standard brewing policy in low end brewers but more precise control of brewing may produce superior results, especially with regard to temperature control.

Policy does not necessarily need to be predisposed to brewing, if the search is guided by a deep artificial network (DNN), with parameters $$\theta$$ randomly initialized and defined as follows.

$$(p, v) = f_{\theta}(s)$$

The brewing process parameters comprise state $$s$$. The vector of control change probabilities $$p$$ have components $$p_a = \Pr(a|s)$$ for each action $$a$$. A scalar $$v$$ estimates the value of the expected outcome $$o$$ from control state $$s$$, where \$v \approx \mathbb{E} [o|s].

Rather than characterize the variations in brewing stock as disturbance $$D$$, it is more consistent with most policy learning approaches to consider the measured properties of brewing stock as features $$\vec{b}$$ and feed a normalized signal $$\mathcal{N}(b)$$ into additional inputs to the deep network. This way, sensitivity to variations in $$\vec{b}$$ is learned as a critical factor in brewing policy.

In such a case, a vanilla MCTS (Monte-Carlo tree search algorithm) can be employed. Using a deep network removes the need to extract domain specific knowledge or discover heuristics to effectively guide control change policy.

Control values must be made discrete, based on granularity that fully (but not excessively) covers brewing possibilities. The control state probabilities that guide the search can be learned using coffee drinker feedback collected in any number of ways, depending the relative costs of each form of outcome data acquisition.

• A focus group

• A group of independent testers

• Repurchase statistics at a selected set of coffee shops

• Feedback controls at a set of offices that provide employees with coffee

For the comprehensiveness in control response to variations in brewing stock properties, the distribution of component values in $$\vec{b}$$ and their statistical correlation over time may need to be controlled during the training process such that trends and distributions resemble likely real world variations in brewing stock.

Each search during the training process is guided by MCTS control changes, $$a_t ∼ \pi_t$$. The final state at the end of training is $$s_T$$, rated using a fixed scale of taster feedback. Selection of each state $$s$$ can be based on a control operation search advantage based on three characteristics.

• Low past experience (a low count of previous visits in the search)

• High probability $$p$$

• High average of values $$\{v, \; ...\}$$ over the leaf states of brewing trials that selected $$a$$ from $$s$$ according to the converging neural network parameters $$f_{\theta}$$

The output of the search is $$\vec{\pi}$$ representing a probability distribution over control changes. The deep network parameters $$\theta$$ converge via gradient descent using a loss function $$\ell$$.

$$\ell = {(o − v)}^2 − {\pi}_T \, \ln{p} + c \, {||\theta||}^2$$

Loss is thus an aggregation of two optimality criteria.

• Mean square estimated disparity between the predicted outcome $$v_t$$t and the taste outcome $$o$$

• Cross-entropy of policy vector $$p_t$$ and search probabilities $${\pi}_t$$

• Hyper-parameter $$c$$ weighted L2 regularization

The updated parameters are used in subsequent brew and taste trials. All control operations must be included in each trial. Success using this approach has been obtained using either greedy or proportionality with respect to the root visit counts.