# Reinforcement Learning with asynchronous feedback

I want suggestions on literature on Reinforcement Learning algorithms that perform well with asynchronous feedback from the environment. What I mean by asynchronous feedback is, when an agent performs an action it gets feedback(reward or regret) from the environment after sometime not immediately. I have only seen algorithms with immediate feedback and asynchronous updates. I don't know if literature on this problem exists. This is why I'm asking here.

My application is fraud detection in banking, my understanding is when a fraud is detected it takes 15-45 days for the system to flag it as a fraud sometimes until the customer complains the system doesn't know its fraud. How would I go about designing a real time system using reinforcement learning to flag transactions that are fraud or normal. Maybe my understanding is wrong, I'm learning on my own if someone could help me I would be grateful.

EDIT: The reason I'm looking at reinforcement learning instead of supervised learning is, its hard to get ground truth data in the banking scenario. Fraudsters are always up-to-date or exceeding the state of the art in fraud detection. So I've decided that reinforcement learning would be an optimal direction to look for solutions to this problem.

• Could you clarify what action the agent is taking? Simply classifying a transaction as fraudulent or not is not really an action - in that it has no influence on any state. For instance, does a transaction marked as fraudulent get reversed automatically (is that what the 15-45 days delay is for?). Also, what is the reward set up? – Neil Slater Jul 30 '18 at 10:57
• I have started the project last week haven't decided on what approach should I take, I have studied RL in my uni so I thought RL would be the optimal way but I didn't find any literature maybe I didn't search using the right keywords. Maybe bandit setup is the right way to go for my problem, since I can't think about how to model this problem with state in it. That's also a reason why I have asked here. Coming to the delay when the system classifies a transaction as genuine and in reality its fraudulent it generally takes 15-45 days due to several reasons. I haven't thought about the reward. – papabiceps Jul 30 '18 at 11:15
• @NeilSlater If we assume that classifying a transaction as fraudulent means blocking it, that can affect the "internal state" of the particular card holder. In the case of a True Positive, this may incentivize the fraudster to leave or to adapt. In the case of a False Positive, this may annoy a genuine customer and incentivize them to leave for a different company if it happens too often. So, in some sense, every individual card holder / account holder is a separate instance of an MDP. – Dennis Soemers Jul 30 '18 at 11:51
• This does lead to concurrent RL problems... which is a very interesting topic that I'm currently trying to wrap up and submit a paper about :D – Dennis Soemers Jul 30 '18 at 11:52
• @DennisSoemers: That is what I am asking, whether the transaction is blocked or not. Blocking a transaction is clearly an action that affects state. However, neither the question nor papabiceps' comment clarifies whether that is the case for his version of the problem (although I would assume it is the case for your research from your answer and comments) – Neil Slater Jul 30 '18 at 11:54

I have been looking for a while into pretty much precisely the problem you describe (including the same application domain), but haven't been able to find much.

The most obvious, mathematically "correct" solution would be to simply delay your standard Reinforcement Learning update rule (of whatever algorithm you choose to implement) by 45 days; if it still wasn't reported as a fraud then, assume it was genuine. This leads to some problems though;

• Need lots of memory to store experiences that were not yet used for updates
• Learning only starts after a significant delay, in which you don't learn anything at all yet and likely therefore run a suboptimal policy for a long time
• Very slow to adapt to new strategies of the fraudsters
• What to do with people who already report fraud cases earlier, like after 10 days? Delay them for the full 45 days anyway, or trigger updates immediately (and potentially mess up the ordering in which experiences actually occurred)?

A quick and dirty "solution" is the following;

• When a transaction occurs, immediately trigger a learning update under the assumption that it was a genuine transaction (for example, with a reward of R = +1).
• If that transaction is later reported as a fraud, trigger an additional update (with same (state, action) pair), but with the negation of the reward that was previously assigned erroneously on top of the normal negative reward for a fraudulent case. For example, if you would normally give R = +1 for genuines, and R = -100 for frauds, give a reward of R = -101 now. This reward will not correct for the previously assigned wrong reward in completely the right way (potentially wrong position in sequence of updates, discounting due to gamma and maybe lambda depending on algorithm used, etc.), but it should be somewhat close (especially if gamma and lambda are close to 1.0).

This is certainly not ideal, has very little theoretical basis and probably breaks quite a bit of Reinforcement Learning theory, but at least it is efficient in terms of computation and memory and in my experience it works alright in practice.

If you're using off-policy RL algorithms, you can use Experience Replay buffers (very popular in DQN-style things in Deep RL these days, but can also be used in tabular RL / RL with linear function approximation etc.). If you already have historical data generated through some non-RL policy in the past (which is typically the case in fraud detection / banking applications, they do have lots of data even if they don't always share it), you can use this to fill your experience replay buffer. In the case of the first solution (at the top of this answer), this can be used for training during the initial delay of 45 days.

Since you expect there to be concept drift though (fraudsters adapting their behaviour over time), you'll want to be careful with experience replay. Old data will become less useful.

A very different style of solution is to assume that you have a team of human experts available who can investigate a very small portion of incoming transactions relatively quickly. This tends to be true for large companies in practice ("investigating" often means a phone-call to a card holder). This enables you to generate accurate feedback for a small portion of your data more quickly, so that you can also do Reinforcement Learning with much less of a delay (albeit only on a small percentage of your experience).

Apart from that idea you might furthermore find it interesting for references to other related work, links to data you could use, etc.

I feel like it should be possible to extend the existing Reinforcement Learning theory with proper algorithms that can properly;

1. Take immediate learning steps with an assumed, default, potentially incorrect reward, and
2. Retroactively correct for previous incorrect updates if the reward turns out to be something else than previously assumed in hindsight.

I'm not aware of existing literature in which this is done though, and it certainly doesn't seem trivial; it will require starting pretty much from "first principles" (e.g., Bellman operator).

Intuitively, I also expect doing this completely correctly will always require a significant amount of memory (memory of all previous transactions of a card holder, such that state-action pairs can be re-generated if necessary). Banks likely already store that kind of data anyway for every customer, so it may not be a problem in practice.

If anyone's planning to work on this, feel free to contact me, I'll likely be happy to collaborate :D

• Thank you very much for the info and pointers. Can you also include how you would go about modelling the problem statement to include state for the fraud detection scenario in your answer. I'm currently reading your paper. I will also write to the professor at my uni and ask his opinion. But this looks like a very good project to do. I think the non-existence of research in this direction is because of the game environments designed as test benches for algorithms. And thank you once again. – papabiceps Jul 30 '18 at 11:25
• @papabiceps Tabular RL will not work, you will need function approximation, with states represented as feature vectors. There has been a lot of research using standard supervised learning for fraud detection, and those also all need feature vectors, so there has been a lot of research into feature engineering as well. Those same kinds of features can be useful here. Some of the papers in this repository are specifically about feature engineering (not all though) – Dennis Soemers Jul 30 '18 at 11:38

That this question uses the word Feedback and made reference to more than one channel of feedback, "Reward and regret," indicates a comprehension of corrective signaling. Some of the reinforcement learning literature that appears scientific lacks that understanding, so beware of that.

The temporal delay of fed back information is not unique to the case of banking fraud detection. It is central to security breach detection in general, including web site hosting and telecommunications hacking. It is also central to many other technology domains ranging from cyber-combat to chemical engineering to petroleum exploration.

Early control systems were of the PID form used in speed or direction governors. In those, temporal elements were only analyzed to avert oscillation, overshoot, and undershoot. Those are still relevant in fraud detection systems, but there are more requirements on the control system, specifically non-linearity in multiple dimensions.

Consequently, control theory has been extended more in the direction of measuring behavioral wellness. Early temporal elements in digital systems included random access memory for applications and persistent memory for programs and data. With the emergence of production ready AI, the temporal elements include acquired rules, fuzzy rule weights, convergence of network parameters corresponding to machine learning components, and other learned information.

The proof of concept in financial fraud detection is the same as for many other domains where the feedback can occur minutes, hours, days, or months after a decision was made or a signal propagated through an artificial learning network: The neural networks of higher life forms, where asynchronous adaptation extends DNA based evolutionary adaptation, pain feedback is augmented by more abstract forms of feedback. In humanoid and primate species, social satisfaction involves a specific signaling that involves neuro-compounds such as serotonin and oxytocin.

This kind of adaptation fits in asynchronicity between reflex and DNA adaptation, in the realm that ranges from Pavlov's conditioned response to the social phenomenon of commitment. The importance of these capabilities is a result of the fact that not all sensory input that provides useful feedback about a behavior exhibited by the biological or artificial control system immediately after it is exhibited.

There is some suggested reading below, and you may want to examine Bayes' Theorem and some of the software you can download in nearly every common programming language that implements what is called Naive Bayesian Categorization. It is through the mathematics of probability theory that the best causal models can be realized. What you probably want to do is learn the key elements of modeling causality with numbers FIRST and then consider how basic probabilistic causality modeling might be augmented with artificial networks.

Although Richard Sutton and Andrew Barton's Reinforcement learning: An introduction (1998 MIT Press) is considered an excellent overview, the early comparative works provide a more direct path to answer questions about algorithms.

When you embark on algorithm development that involves both learning and asynchronicity, it is important to know at the onset that real time programming, such as is now used in high speed trading, is not for the faint at heart. Real time processing places two reliability centered requirements on algorithms, and they should be addressed stringently if you want a stable, low maintenance system that works.

• State-safe — In machine learning, functions that process feedback must not alter a set of interrelated parameters while in use by the forward propagation of the circuit.
• Re-entrant — In machine learning, an interrupt from an incoming signal and a change of state must not frustrate the intent of the algorithm interrupted upon its resuming.

Regarding attacks to banking systems, there will be escalation. The countermeasures the banks take will be met by the countermeasure of the thieves. It is a game, and the banking industry is wise to employ researchers and engineers that understand that learning is feedback dependent.

You may not find the best final designs in the liturature for this reason. Banks naturally employ nondisclosure agreements (NDAs) to keep attackers from gaining knowledge about defensive strategies through web searches. (If it is on the web, it is probably already hacked.)

As researchers and engineers they employ, we are wise to employ asynchronous feedback and real time learning in fraud detection systems and seek a more informed position to stay ahead of engineers that don't value property rights for anyone but themselves.

Suggested Literature

A Unified Analysis of Value-Function-Based Reinforcement-Learning Algorithms, Csaba Szepesvari, Michael L. Littman, October 27, 1998

Asynchronous Methods for Deep Reinforcement Learning, Volodymyr Mnih et al, University of Montreal, 2016

Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates, Shixiang Gu, Ethan Holly, Timothy Lillicrap, Sergey Levine, 2016

Dynamic causal modelling, K.J. Friston, L. Harrison, and W. Penny, Institute of Neurology, UK, 2003

• Reward and regret refer to same thing. reward = - regret. – papabiceps Jul 30 '18 at 12:15
• I think what I want to solve is different from what the papers on asynchronous updates are doing. – papabiceps Jul 30 '18 at 12:36
• Not that simple. Human feedback from fastest to slowest: (a) Reflex, (b) Pain-conditioned response, (c) Reproductive incentive, (d) Social satisfaction -- Regret is in the domain of d. -- The neuro-chemical and behavioral evidence opposes the old idea that pain and pleasure are opposites. Pain meds block pain feedback before its perception and the subject verbalizes it as pleasure. Sexual satisfaction is an independent system that takes minutes. Pain can arise in less milliseconds when a needle pierces the dermis. Pain and reproductive incentive can operate simultaneously. – FauChristian Jul 30 '18 at 12:37
• What were you thinking you want to do? Explain more and maybe I can help. I've done fraud detection for two contracts with a major financial organizations. – FauChristian Jul 30 '18 at 12:39
• In those papers they exploit the parallel nature of multiple threads and use experience replay, capturing an agent’s data which can subsequently be batched and/or sampled over different time-steps. This is similar to batch updates we do in normal batched deep learning. – papabiceps Jul 30 '18 at 12:45