Is it possible to have a fixed trajectory size in the vanilla policy gradient algorithm?

Question

In the concept of the vanilla policy gradient algorithm, is it possible for our trajectory size to be fixed?

For example, my environment is the space of embedded images (using a pre-trained encoder to take images in a space with smaller dimensions), the action I am performing is clustering via k-means algorithm and the reward is the silhouette score metric applied on the clustered images.

I am thinking to set batches of size 100 (dataset is MNIST and its trainset size is 60000). Then take the mean of them and consider this as one observation. Then feed this into the policy network to give me its logits which is an array of size 20 (20 discrete actions). These actions tell me the number of k clusters in the k-means clustering algorithm. One value for k is sampled and the k-means algorithm is applied on these 100 images and then reward is calculated.

I can set a constant number for trajectory sizes, for example, 20 and sum the rewards to get the R(trajectory). Is this possible in the context of the RL and policy gradient, or the trajectory's size cannot be fixed? Also, the action that our policy gives us must lead us to the next observation in the environment, but here images are independent of the policy network's parameters.

I wonder if I can utilize RL to implement this. I appreciate any hints.

Answer 1

Trajectory size can be fixed, but in this case problem would be formulated as something similar to the multi-armed bandit problem where there is a single state and a set of actions to choose from. There is no sequential decision making since samples are not correlated, they are picked at random. So, if you take a batch of 20 examples then you would basically have 20 single state trajectories. For each of those trajectories you can calculate policy gradient and average it over the batch size (which would be 20 in this case).