0
$\begingroup$

I have basically automatised the use of an app through the use of OCR and computer vision.

So basically when a word or an image is detected it will perform a certain action.

When that action is successfully completed it will go to the next state.

Now I want to try basically with a more "heuristic" approach and I thought about reinforcement learning. Why? Because I am aiming to build a tool that basically understand automatically what actions to perform in a certain state.

But I have a doubt. Even though I don't need to declare an association like this (it would beat the purpose of deep reinforcement learning or deep learning in general):

  if(state.MENU_VIEW)
    clickManager.clickOnFolder();
    ...

I still need to define the states, the actions and the reward.

Meaning I would need to instruct my app that when the OCR result is "Open Folder" it means the state I am in is MENU_VIEW. I simply wouldn't tell my app what action to perform in a that state. Am I correct?

What I am trying to say is: how exactly could I make it so that the states (and maybe also the actions?) are generated automatically?

The reward in this case scenario would be basically the folder being opened successfully.

$\endgroup$
10
  • $\begingroup$ Welcome to AI Stack Exchange. It's better if the title is more self-contained (e.g. don't refer to "this scenario" and hope people will click through to read the whole body). I will edit it. If you don't like the edit, feel free to adjust further $\endgroup$ Commented May 31 at 7:29
  • $\begingroup$ Could you clarify what is setting the goals of the automation? I.e. how are the multi-step goals decided, and communicated to the agent? $\endgroup$ Commented May 31 at 7:32
  • $\begingroup$ @NeilSlater thank your for the welcome and thank you for adjusting the title, as a matter of fact it was redundant. $\endgroup$
    – zaxunobi
    Commented May 31 at 12:56
  • $\begingroup$ @NeilSlater In the example provided I could basically pass an int from the environment to the agent: -1 if the click is far from the folder icon, 0 is if near and 1 if it's exactly on top of the folder. $\endgroup$
    – zaxunobi
    Commented May 31 at 13:10
  • $\begingroup$ That would seem like a reward signal - something you would send after the agent has taken an action. What I am asking is how does the agent know what its goal is? Is the goal always to open a folder page, or is it more long term than that, and how flexible is the long term goal? I.e. are you trying to train an agent to achieve one specific task where the UI may vary (so it needs to learn where to look in different systems, but always to do the same thing), or are you trying to train the agent to undertake multiple similar tasks, and it needs to somehow infer which one? $\endgroup$ Commented May 31 at 19:02

1 Answer 1

0
$\begingroup$

In the scenario as you present it, reinforcement learning (RL) should work, but you may gain very little by applying it, over simpler search algorithms.

As you explain in comments, each application is separately trained, the task in each case is arbitrary, and there is no way for the agent to apply contextual knowledge to identify likely actions. The only feedback will be task completion. There are no heurtistics available, simply a list of available actions on each step.

In terms of RL problem definition, this is very similar to solving a maze in least number of steps. The problem is episodic (it terminates), so setting a cost per action (negative reward e.g. $r = -1$ on each step) is the simplest way for the agent to get feedback on how it is doing. The highest total reward will represent the shortest path.

Simple maze solvers are often used as toy problems in RL tutorials. You could use pretty much any example of a gridworld maze from such a tutorial as a starting point to make an agent for your problem, and replace the up/right/left/down actions with your more abstract list of text links or menu options. The state representation will depend on application behaviour, but could be as simple as the current list of available actions.

For this final part, I am assuming the following things, implied by the question and comments:

  • The application behaves completely deterministically.
  • The entry point to the application is always the same.
  • The output goal is a single, shortest-possible list of simple actions (labels to select from) that reach the goal from the start.

If all the above are true, then RL is effectively a random search for the goal state followed by further random iterations to remove non-necessary steps and find the shortest path. In which case, other tree-searching algorithms may be simpler to implment and have better performance than RL. You could try depth-first search (DFS) and/or breadth-first search (BFS), perhaps also requiring loop detection/avoidance. These would avoid the randomness inherent in an RL-based search, and likely be far more efficient due to that.

$\endgroup$
6
  • $\begingroup$ Thanks for the answer! Since I may gain very little by applying it, do you think in this scenario would be preferable a deep learning algorithm (maybe DQNs)? Or should I just stick with the OCR and computer vision approach? $\endgroup$
    – zaxunobi
    Commented Jun 9 at 8:13
  • $\begingroup$ @zaxunobi Deep learning will only help in similar scenarios to RL, when your automation scenarios have something in common and when something is known about the goal task when starting each one. The combination with OCR/vision is a different part of the puzzle. Potentially there is something worth doing there with ML. In the question you presented this as an existing component, but learning to click on meaningful components based on a screenshot might be learnable, separate to finding a correct path to end point $\endgroup$ Commented Jun 9 at 9:22
  • $\begingroup$ Since the actions are pretty much clicking on buttons, what contextual knowledge could an agent apply to determine meaningful components (the action is always a click, what changes are the coordinates of the click)? About the deep learning scenario, do you have in mind how it could learn to to click on meaningful components based on screenshots? $\endgroup$
    – zaxunobi
    Commented Jun 11 at 17:24
  • $\begingroup$ @zaxunobi I don't know enough about your goals and constraints to answer that, and definitely not in the comments. I find it odd that you are writing an automation tool without information on automation goals at the start, but only as a discoverable reward. If you want to explore options for deep RL, I suggest asking a new question giving more details of the project than you have so far $\endgroup$ Commented Jun 11 at 18:23
  • $\begingroup$ The main goal is reaching a certain state of the program, or the "end" if you like. The reason why I am going with "discoverable reward" it's because the only information that I could provide it's basically (which I am doing with OCR already) is that if a certain text is detected then I am in a specific STATE and therefore will be performed a specific ACTION. But doing this would defeat the whole point of using RL, don't you think? $\endgroup$
    – zaxunobi
    Commented Jun 12 at 6:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .