I don't know what to search for, so I'll try to describe what I'm trying to do : I want an AI that learns by showing it how to navigate on the Internet.

Let's say I want to save 50 images from a website.

1) what do you call this style of learning : I show the AI 1 to 5 saves, and it has to do the next

2) does it exist ? Is it possible to go "this far" ? In particular, the AI has to learn how to deal with connection errors (repeat a bit later for instance).

3) should it be based on the code of the page, or a screenshot, or something else ?

Thank you in advance, I need the english terms for further researchs, and leads for algorithms (I want it to work also in Windows, for renaming files) ^^

  • $\begingroup$ The described problem of automatically downloading images from the internet isn't called a “Learning from demonstration” (LfD) task. In the literature, LfD is defined as “finding the reward function by a human operator with guided policy search”. In contrast, the aim to download images from the internet, is a normal scripting problem which can be solved by tools like wget and a textfile which contains all the urls. $\endgroup$ – Manuel Rodriguez Dec 18 '18 at 9:42
  • $\begingroup$ Downloading images was just a simple example. The aim is to build an AI that can mimic a serie of actions it is shown once (or at most let's say 10 times). For instance, moving files and renaming them, go to some website and do some actions (like registering me, downloading some images, saving some titles in a notepad for later use...). Of course, I can program all these actions with scripts, but AI is more versatile, and it is its point ^^ $\endgroup$ – lrosique Dec 18 '18 at 12:24

One term that would describe it is, "Automated navigation," but no one can easily guess what others might call it. The requirement is a little fuzzy, which doesn't help name it properly. We can go over a few of the more likely variants.

  • The system learns a particular navigation path, in which the case is called, "Test Case Recording," because functional tests are constructed using Selenium, Watir, and other tools. That's not AI though.
  • The AI system naively learns how a particular person or set of people navigate when performing a particular task. Then it could be called, "Goal oriented web browsing automation." The naive learning of a goal oriented browsing pattern is fairly easy to do.
  • The AI system cognitively learns how a particular person or set of people navigate when performing a particular task. Some level of AI comprehension of choices would requires a self-constructing model and significant research, possibly years or decades of it.
  • The AI system learns how a particular person or set of people navigate when they don't have a defined goal in mind. Then it could be called, "Stream of consciousness browsing automation," however I can't see any purpose in that.
  • The AI system seeks particular documents, in which case it would be a, "Network node classifier."

It is unlikely that demonstrating 1 to 5 saves would be sufficient unless it is the first item in the above list, which isn't AI. It's recording and putting a loop around like streams. Now if the AI is not involved in the recording at all but is to intelligently create the loop from two or three like sequences and know what to put before, inside, and after the loop to navigate in, iterate, and then finalize the task, perhaps by logging out, that is an offline task. That could be called, "Loop Configuration Automation," or, "Automated Iteration Programming."

Dealing with connection errors by waiting and trying again is not necessarily AI either, unless the wait period is driven by the probability distribution of success after waits based on past experience.

Screenshots are a bad idea. Look into Selenium, Katalon, and Watir. You want to be able to record and play back, even if the play back is generated by the AI instead of simply playing the recorded sequence.

These high level requirements will not be helpful in finding algorithms. To get into the algorithms, one would first have to clarify the high level requirements. Whether an MLP, LSTM, RBM, DQN, CNN, MDP, or whatever is appropriate can be determined when a clear understanding of exactly what it is that drives the search as an objective is known. If it is not know, no value or loss functions can be constructed intelligently and no solution is likely to arise.

Addendum in Response to Question Author Comment

The AI component would be receiving all the pertinent information regarding the HTTP(S) transaction during the training. If using Selenium, Katalon, Watir, or some other functional browsing tool, the pertinent information would be the DOM model and the events trapped by the browser from the mouse and keyboard or touch screen.

After learning completes, the AI component would still take as input the HTTP(S) request or DOM (depending on which layer of abstraction was used for training) and then drive that same layer using the trained component.

Using the higher level DOM will improve reliability significantly (possibly by a factor of thousands or more) because the AI will deal with web documents, forms, and events at the actual user interface, but DOM interaction is slower than dealing directly with the HTTP(S) request and response. Training and using the training using the DOM will require what is called a headless browser, but both Chrome and FireFox can operate that way in UNIX or LINUX with Xvfb.

The largest task in the project is to do thorough enough example log ins to learn. It may take many, and the It has to find the right link (navigation), recognize the form and fill it.

Finding the website names of all your tabs in chrome is not an AI task. There are plenty of browser plug-ins that can do that.

| improve this answer | |
  • $\begingroup$ First of all, thank you for your response and all your keywords, that's perfect ^^ Now, some questions : where do you place the AI that receives an URL and has for mission to register me on the website (like I give it facebook.com) ? It has to find the right link (navigation), recognize the form and fill it. Then validate, and validate the validation (or correct the inputs if there is an error). I don't take into account the captcha. And an AI that can find the website names of all your tabs in chrome ? (that's just an example to help me fix my ideas ^^). Thanks :) $\endgroup$ – lrosique Dec 18 '18 at 12:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.