# How does reinforcement learning handle measured disturbances?

I recently encountered an interesting problem and was wondering how RL would solve it. The objective of the problem is to maximize the coffee quality, given by box X. The coffee quality objective function is defined by the company.

To maximize the quality of the coffee, we can perform 2 actions:

• change stirring speed of the coffee machine
• change the temperature of the coffee machine

Now to the tricky part. The coffee bean characteristic coming into the coffee machine is random. We can measure its characteristics before sending them to the coffee machine, but we cannot change them.

I formulated this problem into a control problem, such that $$X$$ is a function of my previous states, $$X(t - k)$$, the control input , $$U$$, and the measured disturbance, $$D$$. Given a constant $$D$$, the problem is trival to solve because the disturbance is a constant part of the environment. However, during times when $$D$$ changes rapidly, the policy is no longer optimal.

How do I inject the information of the measured disturbance into my RL agent?

There are two ingredients in the task. The first one is a forward model which is wrong, because noise / disturbance has reduced it's prediction quality. And the second is a reinforcement learning algorithm for optimal control of this system. Let us describe, what exactly a modeling error is. It's a mismatch between the real system and the simulation. The deviation can be measured. On the time-graph there is a gap between the predicated state and the real state. That means, the system is aware that something is wrong.

The second problem is how to make the controller “fault-tolerant”. That means, in the case there is a gap, which control signal is needed. A naive solution would be to track many forward models at the same time and to produce control signals for all of them. Another idea is to adjust the forward model on the fly. That means, in case of disturbance, a new forward model is created and for this single one the control signal is generated. The option three is more easier to implement but it's the best-practice method for real control systems. If the measured disturbance is too high, the system prints out an error and a human operator has to take over. This solution can't be called reinforcement learning, but error detection. That means, the disturbance is measured and if its above the threshold an error an error message is dropped.

• Hi Manuel, thanks for the reply. However, your solution seems to be saying that we can only build a RL that can handle the most probabilistic coffee bean quality, and the moment the coffee bean quality changes, RL does not work. I don't believe that is a solution for a company, because that is basically saying that RL is completely useless in this case. One way we could formulate the problem is a Contextual bandit problem, where given the disturbance (inlet composition of coffee beans), we just make the optimal decisions, and then reward is based on the output coffee quality. – Rui Nian Dec 15 '18 at 18:10
• But that is not RL, and does not work great in continuous systems. In advanced controls like model predictive control and LQR, the disturbance can be directly injected into the optimization problem, and such problems are trivial to solve. So I am trying to find a way for RL to also be "injected" with the disturbance while making decisions. – Rui Nian Dec 15 '18 at 18:11

Use of reinforcement learning makes most sense over LSTM if control parameters can change during brewing to optimize results, which is not standard brewing policy in low end brewers but more precise control of brewing may produce superior results, especially with regard to temperature control.

Policy does not necessarily need to be predisposed to brewing, if the search is guided by a deep artificial network (DNN), with parameters $$\theta$$ randomly initialized and defined as follows.

$$(p, v) = f_{\theta}(s)$$

The brewing process parameters comprise state $$s$$. The vector of control change probabilities $$p$$ have components $$p_a = \Pr(a|s)$$ for each action $$a$$. A scalar $$v$$ estimates the value of the expected outcome $$o$$ from control state $$s$$, where \$v \approx \mathbb{E} [o|s].

Rather than characterize the variations in brewing stock as disturbance $$D$$, it is more consistent with most policy learning approaches to consider the measured properties of brewing stock as features $$\vec{b}$$ and feed a normalized signal $$\mathcal{N}(b)$$ into additional inputs to the deep network. This way, sensitivity to variations in $$\vec{b}$$ is learned as a critical factor in brewing policy.

In such a case, a vanilla MCTS (Monte-Carlo tree search algorithm) can be employed. Using a deep network removes the need to extract domain specific knowledge or discover heuristics to effectively guide control change policy.

Control values must be made discrete, based on granularity that fully (but not excessively) covers brewing possibilities. The control state probabilities that guide the search can be learned using coffee drinker feedback collected in any number of ways, depending the relative costs of each form of outcome data acquisition.

• A focus group

• A group of independent testers

• Repurchase statistics at a selected set of coffee shops

• Feedback controls at a set of offices that provide employees with coffee

For the comprehensiveness in control response to variations in brewing stock properties, the distribution of component values in $$\vec{b}$$ and their statistical correlation over time may need to be controlled during the training process such that trends and distributions resemble likely real world variations in brewing stock.

Each search during the training process is guided by MCTS control changes, $$a_t ∼ \pi_t$$. The final state at the end of training is $$s_T$$, rated using a fixed scale of taster feedback. Selection of each state $$s$$ can be based on a control operation search advantage based on three characteristics.

• Low past experience (a low count of previous visits in the search)

• High probability $$p$$

• High average of values $$\{v, \; ...\}$$ over the leaf states of brewing trials that selected $$a$$ from $$s$$ according to the converging neural network parameters $$f_{\theta}$$

The output of the search is $$\vec{\pi}$$ representing a probability distribution over control changes. The deep network parameters $$\theta$$ converge via gradient descent using a loss function $$\ell$$.

$$\ell = {(o − v)}^2 − {\pi}_T \, \ln{p} + c \, {||\theta||}^2$$

Loss is thus an aggregation of two optimality criteria.

• Mean square estimated disparity between the predicted outcome $$v_t$$t and the taste outcome $$o$$

• Cross-entropy of policy vector $$p_t$$ and search probabilities $${\pi}_t$$

• Hyper-parameter $$c$$ weighted L2 regularization

The updated parameters are used in subsequent brew and taste trials. All control operations must be included in each trial. Success using this approach has been obtained using either greedy or proportionality with respect to the root visit counts.