Reinforcement Learning with asynchronous feedback

Question

I want suggestions on literature on Reinforcement Learning algorithms that perform well with asynchronous feedback from the environment. What I mean by asynchronous feedback is, when an agent performs an action it gets feedback(reward or regret) from the environment after sometime not immediately. I have only seen algorithms with immediate feedback and asynchronous updates. I don't know if literature on this problem exists. This is why I'm asking here.

My application is fraud detection in banking, my understanding is when a fraud is detected it takes 15-45 days for the system to flag it as a fraud sometimes until the customer complains the system doesn't know its fraud.

How would I go about designing a real-time system using reinforcement learning to flag transactions that are fraudulent or normal?

Maybe my understanding is wrong, I'm learning on my own if someone could help me I would be grateful.

The reason I'm looking at reinforcement learning instead of supervised learning is, it's hard to get ground truth data in the banking scenario. Fraudsters are always up-to-date or exceeding the state of the art in fraud detection. So I've decided that reinforcement learning would be an optimal direction to look for solutions to this problem.

Answer 1

I have been looking for a while into pretty much precisely the problem you describe (including the same application domain), but haven't been able to find much.

The most obvious, mathematically "correct" solution would be to simply delay your standard Reinforcement Learning update rule (of whatever algorithm you choose to implement) by 45 days; if it still wasn't reported as a fraud then, assume it was genuine. This leads to some problems though;

• Need lots of memory to store experiences that were not yet used for updates
• Learning only starts after a significant delay, in which you don't learn anything at all yet and likely therefore run a suboptimal policy for a long time
• Very slow to adapt to new strategies of the fraudsters
• What to do with people who already report fraud cases earlier, like after 10 days? Delay them for the full 45 days anyway, or trigger updates immediately (and potentially mess up the ordering in which experiences actually occurred)?

---

A quick and dirty "solution" is the following;

• When a transaction occurs, immediately trigger a learning update under the assumption that it was a genuine transaction (for example, with a reward of R = +1).
• If that transaction is later reported as a fraud, trigger an additional update (with same (state, action) pair), but with the negation of the reward that was previously assigned erroneously on top of the normal negative reward for a fraudulent case. For example, if you would normally give R = +1 for genuines, and R = -100 for frauds, give a reward of R = -101 now. This reward will not correct for the previously assigned wrong reward in completely the right way (potentially wrong position in sequence of updates, discounting due to gamma and maybe lambda depending on algorithm used, etc.), but it should be somewhat close (especially if gamma and lambda are close to 1.0).

This is certainly not ideal, has very little theoretical basis and probably breaks quite a bit of Reinforcement Learning theory, but at least it is efficient in terms of computation and memory and in my experience it works alright in practice.

---

If you're using off-policy RL algorithms, you can use Experience Replay buffers (very popular in DQN-style things in Deep RL these days, but can also be used in tabular RL / RL with linear function approximation etc.). If you already have historical data generated through some non-RL policy in the past (which is typically the case in fraud detection / banking applications, they do have lots of data even if they don't always share it), you can use this to fill your experience replay buffer. In the case of the first solution (at the top of this answer), this can be used for training during the initial delay of 45 days.

Since you expect there to be concept drift though (fraudsters adapting their behaviour over time), you'll want to be careful with experience replay. Old data will become less useful.

---

A very different style of solution is to assume that you have a team of human experts available who can investigate a very small portion of incoming transactions relatively quickly. This tends to be true for large companies in practice ("investigating" often means a phone-call to a card holder). This enables you to generate accurate feedback for a small portion of your data more quickly, so that you can also do Reinforcement Learning with much less of a delay (albeit only on a small percentage of your experience).

You can read more about this idea in the following paper (disclaimer: I'm an author on it):

• Dennis J.N.J. Soemers, Tim Brys, Kurt Driessens, Mark H.M. Winands, and Ann Nowé (2018). “Adapting to Concept Drift in Credit Card Transaction Data Streams Using Contextual Bandits and Decision Trees”. In Thirtieth Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-18), pp. 7831-7836. AAAI Press.

Apart from that idea you might furthermore find it interesting for references to other related work, links to data you could use, etc.

---

I feel like it should be possible to extend the existing Reinforcement Learning theory with proper algorithms that can properly;

1. Take immediate learning steps with an assumed, default, potentially incorrect reward, and
2. Retroactively correct for previous incorrect updates if the reward turns out to be something else than previously assumed in hindsight.

I'm not aware of existing literature in which this is done though, and it certainly doesn't seem trivial; it will require starting pretty much from "first principles" (e.g., Bellman operator).

Intuitively, I also expect doing this completely correctly will always require a significant amount of memory (memory of all previous transactions of a card holder, such that state-action pairs can be re-generated if necessary). Banks likely already store that kind of data anyway for every customer, so it may not be a problem in practice.

If anyone's planning to work on this, feel free to contact me, I'll likely be happy to collaborate :D

Answer 2

That this question uses the word Feedback and made reference to more than one channel of feedback, "Reward and regret," indicates a comprehension of corrective signaling. Some of the reinforcement learning literature that appears scientific lacks that understanding, so beware of that.

The temporal delay of fed back information is not unique to the case of banking fraud detection. It is central to security breach detection in general, including web site hosting and telecommunications hacking. It is also central to many other technology domains ranging from cyber-combat to chemical engineering to petroleum exploration.

Early control systems were of the PID form used in speed or direction governors. In those, temporal elements were only analyzed to avert oscillation, overshoot, and undershoot. Those are still relevant in fraud detection systems, but there are more requirements on the control system, specifically non-linearity in multiple dimensions.

Consequently, control theory has been extended more in the direction of measuring behavioral wellness. Early temporal elements in digital systems included random access memory for applications and persistent memory for programs and data. With the emergence of production ready AI, the temporal elements include acquired rules, fuzzy rule weights, convergence of network parameters corresponding to machine learning components, and other learned information.

The proof of concept in financial fraud detection is the same as for many other domains where the feedback can occur minutes, hours, days, or months after a decision was made or a signal propagated through an artificial learning network: The neural networks of higher life forms, where asynchronous adaptation extends DNA based evolutionary adaptation, pain feedback is augmented by more abstract forms of feedback. In humanoid and primate species, social satisfaction involves a specific signaling that involves neuro-compounds such as serotonin and oxytocin.

This kind of adaptation fits in asynchronicity between reflex and DNA adaptation, in the realm that ranges from Pavlov's conditioned response to the social phenomenon of commitment. The importance of these capabilities is a result of the fact that not all sensory input that provides useful feedback about a behavior exhibited by the biological or artificial control system immediately after it is exhibited.

There is some suggested reading below, and you may want to examine Bayes' Theorem and some of the software you can download in nearly every common programming language that implements what is called Naive Bayesian Categorization. It is through the mathematics of probability theory that the best causal models can be realized. What you probably want to do is learn the key elements of modeling causality with numbers FIRST and then consider how basic probabilistic causality modeling might be augmented with artificial networks.

Although Richard Sutton and Andrew Barton's Reinforcement learning: An introduction (1998 MIT Press) is considered an excellent overview, the early comparative works provide a more direct path to answer questions about algorithms.

When you embark on algorithm development that involves both learning and asynchronicity, it is important to know at the onset that real time programming, such as is now used in high speed trading, is not for the faint at heart. Real time processing places two reliability centered requirements on algorithms, and they should be addressed stringently if you want a stable, low maintenance system that works.

• State-safe — In machine learning, functions that process feedback must not alter a set of interrelated parameters while in use by the forward propagation of the circuit.
• Re-entrant — In machine learning, an interrupt from an incoming signal and a change of state must not frustrate the intent of the algorithm interrupted upon its resuming.

Regarding attacks to banking systems, there will be escalation. The countermeasures the banks take will be met by the countermeasure of the thieves. It is a game, and the banking industry is wise to employ researchers and engineers that understand that learning is feedback dependent.

You may not find the best final designs in the liturature for this reason. Banks naturally employ nondisclosure agreements (NDAs) to keep attackers from gaining knowledge about defensive strategies through web searches. (If it is on the web, it is probably already hacked.)

As researchers and engineers they employ, we are wise to employ asynchronous feedback and real time learning in fraud detection systems and seek a more informed position to stay ahead of engineers that don't value property rights for anyone but themselves.

Suggested Literature

A Unified Analysis of Value-Function-Based Reinforcement-Learning Algorithms, Csaba Szepesvari, Michael L. Littman, October 27, 1998

Asynchronous Methods for Deep Reinforcement Learning, Volodymyr Mnih et al, University of Montreal, 2016

Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates, Shixiang Gu, Ethan Holly, Timothy Lillicrap, Sergey Levine, 2016

Dynamic causal modelling, K.J. Friston, L. Harrison, and W. Penny, Institute of Neurology, UK, 2003