So solution 2 is like creating an auto encoder trained on all images and then concatenating the latent vectors as input to the head?

How would you continue training with PPO afterwards? If you setup a DQN to do the imitation learning part then how do you copy those parameters into the PPO network. I think I am able to generate a good amount of pretty good trajectories for my problem. What would be the difference between using those in a DQN network versus using the imitation learning you have suggested?