I am re-implementing vpg and using Spinning Up as reference implementation. I noticed that the default epoch size is 4000. I also see cues in papers that big batch size is quite standard.
My implementation doesn't batch XP together just applies the update after every episode. It turns out my implementation is more sample efficient than the reference implementation on simple problems (like CartPole or LunarLander) even though I haven't added the critic yet! Of course this could be due to number of reasons, for example I've only done parameter search on my implementation.
But it would make sense anyway: bigger batch size is generally considered better only because the GPU is faster processing many samples parallel. Is this the reason here? It would make sense but it is surprising for me as I thought sample efficiency is considered more important than computing efficiency in RL.