# What are good action outputs for reinforcement learning agents acting in a trading environment?

I am trying to build an agent that trades commodities in a exchange setting. What are good ways to map the action output to real world actions? If the last layer is a tanh activation function, outputs range between [-1,+1]. How do I map these values to real actions? Or should I change the output activation to linear and then directly apply the output as an action?

So let's say the output is tanh activated and it's -0.4, 5. I could map this to: - -0.4 --> sell 40% of my holdings for 5$per unit - -0.4 --> sell 40% for 5$ in total

if it was linear, I could expect larger outputs (e.g -100, 5). Then the action would be mapped to: - sell 100 units for 5$each - sell 100 units for 5$ total

Working Backward

Working backward from the trading interface available to you Note 1, you will need two things for each Exchange-Traded Fund (ETF) or other tradable commodity of another class.

• An operation to perform
• An associated monetary amount

The system can have an output structure Note 2 for each ETF like this (depending of course on the trading interface available to you to you and your banks primary monetary system).

• Ternary operation indicator (buy, sell, hold)

Why Not Just One Number per ETF?

A few corroborating reasons exist for why a single positive, zero, or negative trade amount is not likely the optimal architectural choice.

• The difference between holding and trading an amount of 1.0 monetary unit is not equivalent to the difference between trading 1.0 and 2.0 monetary units. Stated mathematically, the function of profitability to trade amount is not smooth and probably not even continuous.
• When you implement the ternary output as two binary outputs {buy, sell}, training against the Boolean expression (buy AND sell) is likely to improve your initial performance and possibly your ongoing performance Note 3.

Limitations on Real Trades to Consider

Because you have the limit of the assets in the liquid account from which you can buy, you will need to train against breaking the bank or using a stage after the NN outputs imposing rules or a formula based on gain and loss probabilities. This financial constraint muddies your question because there are several ways to ensure you do not break the bank (get an insufficient funds response from your trade operation).

Optimizing a Deeper Architecture

Let's first assume you use probabilistic calculus to produce a closed form (formula) for how to trade based on predictions from the NN architecture you design. Then the NN outputs might be continuous values representing the distribution of outcomes for each ETF. Projections will almost always be dependent on investment duration.

In such an architecture the NN output activation scheme would be a continuous function (not necessarily linear) producing something like this Note 4.

• Mean expected delta value in one day
• Std deviation in one day expectation
• Mean expected delta four weeks
• Std deviation in two week expectation
• Mean expected delta two years
• Std deviation in two years expectation

Any NN optimization of an investment portfolio that does not inherently deal with probability is nonsense. Optimizing for maximum gain will introduce great risk. Optimizing for minimum risk could result in losses. The goal of optimization must be some representation of the balance between the desire to win and the fear of loosing, to put it in anthropological terms.

Mean and standard deviation are obvious starting choices, and the traditional categories of short, medium, and longer term investment is also reasonable to begin with in the temporal domain Note 5.

Pure NN

Now let's assume you replace the calculus with another NN scheme trained to maximize portfolio total assets Note 6. Such a replacement NN scheme must also take as an input your available liquid asset amount along with the above probabilistic projections. You must also train to ensure the aggregation of buys and sells do not break your bank, that your liquid asset account never drops below zero.

The trade amount activation should be a continuous function, but not necessarily linear and probably not tanh either because the asymptotes of that function would be counterproductive unless it is made proportional to your available liquid assets to train by aggregating your options in your architecture. However, that's not optimal because you may find a better deal to use those assets a minute later.

The odd roots (third root or fifth root or both with coefficients), when used along with training to not break the bank and to maximize the rate of portfolio growth will produce a better environment for learning in the earlier layers because of the probabilistic and aggregation realities of liquid asset limitations.

NOTES

Note 1 — Preferably a secure RESTful API, however experimentation can employ a web browser the HTTPS transactions you can control using https://github.com/SeleniumHQ/selenium or https://github.com/watir/watir.

Note 2 — Input would be things like number of companies, exposure ratings provided by investment firm(s), short options, inception date, and a sequence of events, each event containing fixed point numbers and flags like these.

• Price per share
• Closing price flag
• Expense ratio
• Dividend per share

Note 3 — The binary output vector {buy, sell} value of {1, 0} causes a buy. The value of {0, 1} causes a sell. The value {0, 0} is ignored, and {1, 1} may trigger another training operation to correct this illegal output, which is likely to be indicative of the staleness of the last round of training. If the NN is re-entrant (reinforced), the feedback vector could include this anomalous flag or weight it heavily in an aggregation of feedback sources. In summary, the ternary scheme can be expected to augment the training speed and resulting accuracy. More importantly, it opens additional options for continuous optimization.

Note 4 — Delta is an aggregation of price increase or loss, dividends, and holding and trading expenses because that is the proper metric related to profitability of the portfolio.

Note 5 — Four weeks and two years have been chosen for the middle and longer range projections so that the time ratios are 26.09 and 28 between the three durations. Common choices are temporally skewed. If 1 day, 1 week, and 1 year had been chosen, the ratios would have been 7.0 and 52.18. If 1 day, 1 month, and 1 year had been chosen, they would have been 30.44 and 12.

Note 6 — Do not assume that even a well trained NN will ever outperform formulae properly derived from probabilistic calculus for the last stage in a trading profitability architecture.