How do I create a custom gym environment based on an image?

I am trying to create my own gym environment for the A3C algorithm (one implementation is here). The custom environment is a simple login form for any site. I want to create an environment from an image. The idea is to take a screenshot of the web page and create an environment from this screenshot for the A3C algorithm. I know the doc and protocol for creating a custom environment. But I don't understand how to create an environment, exactly, based on a screenshot.

If I do so

self.observation_space = gym.spaces.Box(low=0, high=255, shape=(128, 128, 3), dtype=np.uint8)


I get a new pic.

Here's the algorithm that I am trying to implement (page 39 of the master's thesis Deep Reinforcement Learning in Automated User Interface Testing by Juha Eskonen).

• Well, you need to represent the screenshot as an "observation" of your environment. A screenshot is an image, so you can read this image and turn it into an array, which should be your observation. Maybe this code github.com/openai/gym/blob/master/gym/wrappers/… can be useful. In any case, I would like to note that our site focuses on theoretical questions, but this is a programming question. Given that this question is about a RL topic, I will not close it as off-topic, given that RL is central to AI. Check this: ai.stackexchange.com/help/on-topic. – nbro Nov 4 '20 at 11:31
• If you don't want to create the environment, what is your question then? – nbro Nov 4 '20 at 11:48
• No, I want) I just uderstand, I reread the paper many times and algorithm and understand that we use a3c for creating a probability map for each element and then we do some action using third part api, like selenium. – Ren Nov 4 '20 at 11:50
• Ok. But what is your question? It was "how to create this custom environment from screenshots?", right? – nbro Nov 4 '20 at 11:52
• Right) My question was "how to create this custom environment from screenshots?" – Ren Nov 4 '20 at 11:53