For a personal project, I'm trying to download files from a specific set of websites using a web scraper. The scraper has to navigate multiple webpages to get to the files I want to download. I'd like to use AI to find each successive link in order to avoid the inflexibility of hardcoded DOM paths. In other words, I want an AI system that replicates the way that a human would click through links to get to a download page.

This seems like a task for a supervised neural network, where the input would be the HTML page and the output would be the href link to the next webpage in the search. But that's about the extent of my AI knowledge.

Which subtype of neural network would be most effective for this kind of problem?

Note, I have looked at this related question. My problem would probably fall under bullet two.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.