0
$\begingroup$

I have a task which I would like to teach an AI to perform. The input to the task will a screenshot of the screen and the output at any given time step is one of the following actions:

    DoNothing
    Click(x, y)
    Drag(from_x, from_y, to_x, to_y)
    KeyPress(Key)
    LongKeyPress(Key, duration)

where:
    Key ∈ { 1 , 2 , 3 , 4 , Space , Up , Down , Left , Right }

I think that deciding which action to take could be done with a softmax output layer, but I don't know how I would also encode the parameters of each action. For example, the Click action also 2 co-ordinate values but the Drag action needs 4 co-ordinates.

My best idea at the moment is that I could use one neural network to decide on which action type to perform and for each action, train another network to decide the parameters. So for example I have the main network which decides to take a Click action, and then a secondary network which will choose the x and y co-ordinates (given the same input).

The problem with this is that I would end up training 6 networks (decide the action, Click co-ordinates, Drag co-ordinates, KeyPress key, LongKeyPress key, LongKeyPress duration). The drawback for this is that I would need a very large dataset with a lot of samples for each type.

I considered how it might be possible to encode all the values into one output, but I wouldn't know how to deal with missing values (for samples where the user clicked, what value should i use for the Key output of KeyPress).

I have tried to find research papers on the topic, but I must be missing the right keywords because I can't find anything about strategies for this problem. Any ideas or pointers to relevant material would be great!

Improve this question
$\endgroup$
2
  • $\begingroup$ I think your problem would fit to reinforcement learning, but you also mentioned a dataset - so the main question is: Given a screenshot, is there one and only one correct action to perform? If yes, I think you are on the right track with classification, If No you should explore the realm of reinforcement learning (RL). Also, if this is RL, the number of possible actions is enormous (especially the continuous actions click + drag & drop) which makes the task really hard and you might want to simplify the actions a bit. $\endgroup$
    – Chillston
    May 14, 2022 at 7:59
  • $\begingroup$ Unfortunately, I cannot explore the possibility of RL because I do not have a test environment in which the agent can 'practice'. Besides this, I will mention that I am not interested in finding the most efficient solution, but instead the most human-like solution. That is why I would like the agent to learn from a dataset of actions made by a human. I know that I might also be able to use a GAN architecture for this but that is something I have no done before. Also I am aware that the action space is huge, but I cannot see a feasible way to reduce it without also reducing expressiveness. $\endgroup$
    – turbo_sheep
    May 14, 2022 at 12:33

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .