The optimization problem I'm trying to solve has been approximated analytically - the approximation to the optimal solution depends on three variables (t-remaining time, q-current inventory and s-current price). It is precisely those variables that I've included in my state space. Therefore, RL should in theory be able to learn this function, provided enough episodes and an NN function approximator with enough capacity.

OK, clear now, thanks. But another thing just crossed my mind. Let's forget about REINFORCE for a moment. Why is DDPG preferred over simply having a NN (which represents a policy) output a deterministic action, calculating the loss function for this policy (say negative Sharpe ratio or whatever) and simply using the gradient descent to find optimal parameters of the NN? This way we don't have to deal with either stochastic policies or the intricacies of actor-critic approaches.

"That's true, but how do you get to that stage of training with REINFORCE?" Why can't I simply start out with a stochastic Gaussian policy with a very small variance, and then adjust its mean while learning but keep the variance fixed.

"plus have state include the agent's current portfolio of investment and working funds." Yes, this is a common way of (in my opinion artificially) introducing sequentiality (in a sense that now an action at time t will affect the state at time t+1). However, if we ignore the trading costs (which is commonly done in many papers), the optimal action at time t+1 will clearly NOT depend on the part of the state at time t+1 that represents current portfolio weights (as one should invest in stocks that will grow in the future no matter what the current portfolio weights are).

Unfortunately it did not help with the learning error - the average reward still doesn't seem to be converging.