I am currently working on my first research project, which is to implement machine learning based web scraping in the e-recruitement environment. Currently, I am building the dataset that needs to be annotated afterwards. However, I get lost in the different approaches and don't really know which one is most appropriate.
The Goal:
Given an HTML web page with a job posting, the machine learning system (e.g. Natural Language Processing Models) should return the responsibilities and duties, required qualifications, required skills, benefits and some other things mentioned in the job description.
The Data:
40,000 downloaded web pages from 6 different e-recruiting platforms in the United States for 10 computer science/technology job categories. The web pages are downloaded as a single HTML file with CSS, images, ads, and the original layout still present. The plan is to use the Boilerpipe Python port to extract the job description text for subsequent natural language processing tasks. This worked reasonably well for a random sample of the data; with a small dictionary and regular expressions, it should be possible to remove the noise.
Questions:
- What methods do you recommend I use to extract responsibilities and duties, required qualifications, required skills, benefits, and some other things mentioned in the job description from future, possibly unseen job postings in the United States for the above 10 computer science/technology job categories?
- What topics or literature do you recommend for me to use for this endeavor?
