What do the authors of this paper mean by the bias term in this picture of a neural network implementation?

Question

I am reading a paper implementing a deep deterministic policy gradient algorithm for portfolio management. My question is about a specific neural network implementation they depict in this picture (paper, picture is on page 14).

The first three steps are convolutions. Once they have reduced the initial tensor into a vector, they add that little yellow square entry to the vector, called the cash bias, and then they do a softmax operation.

The paper does not go into any detail about what this bias term could be, they just say that they add this bias before the softmax. This makes me think that perhaps this is a standard step? But I don't know if this is a learnable parameter, or just a scalar constant they concatenate to the vector prior to the softmax.

I have two questions:

1) When they write softmax, is it safe to assume that this is just a softmax function, with no learnable parameters? Or is this meant to depict a fully connected linear layer, with a softmax activation?

2) If it's the latter, then I can interpret the cash bias as being a constant term they concatenate to the vector before the fully connected layer, just to add one more feature for the cash assets. However, if softmax means just a function, then what is this cash bias? It must be a constant that they implement, but I don't see what the use of that would be, how can you pick a constant scalar that you are confident will have the intended impact on the softmax output to bias the network to put some weight on that feature (cash)?

Any comments/interpretations are appreciated!

Answer 1

to the cash bias: I think this is simply the money that is still available at time t=50 and has not yet been invested.

Answer 2

1. Yes, the softmax is just a softmax, applied to the 12 values from the final linear layer.

2. As far as the "cash bias" goes — I'm not sure if the "cash" part has significance to the authors. But, the "bias" part is standard. The inputs to the last layer are 11-dimensional vectors. But the last layer has 12 parameters: one parameter for each input value, and one additive parameter called the bias.

That is, assuming the input vector is $(x_1, x_2, ..., x_{11})$, then output will be:

$$ y = w_0 + x_1w_1 + x_2w_2 + ... + x_{11}w_{11} $$

The weight $w_0$ is called a bias. If you remove it, then you'll always map the input of all zeros, $(0, ... 0)$ to the output of $0$. This might not be desirable, and might prevent the layer from converging. Adding a bias $w_0$ allows the layer to map any input to any output.

Not that the $w_0$ is not a true constant. It's a trainable parameter like all others.

Answer 3

On page 8 they say:

In the experiments of the paper, the 11 most-volumed non-cash assets are preselected for the portfolio. Together with the cash, Bitcoin, the size of the portfolio, m + 1, is 12. This number is chosen by experience and can be adjusted in future experiments. For markets with large volumes, like the foreign exchange market, m can be as big as the total number of available assets.

They have 12 possible actions in the policy: buy one or more of the 11 non-cash assets, or sell them and hold more cash. It seems that the bias towards the cash position is a learnable parameter.

Although now that I read that quote a few times, it seems that they treat Bitcoin as cash?! At first I thought they meant a currency like USD. But in effect the outcome is the same. As you increase the cash bias, the softmax activation gets biased towards that policy.