6
$\begingroup$

first of all I want to specify the data available and what needs to be achieved: I have a huge amount of vacancies (in the millions). The information about the job title and the job description of each vacancy are stored separately. I also have a list of professions (around 3000), to which the vacancies shall be mapped.

Example: java-developer, java web engineer and java software developer shall all be mapped to the profession java engineer.

Now about my current researches and problems: Since a lot of potential training data is present, I thought a machine learning approach could be useful. I have been reading about different algorithms and wanted to give neural networks a shot.

Very fast I faced the problem, that I couldn't find a satisfying way to transform text of variable length to numerical vectors of constant size (needed by neural networks). As discussed here, this seems to be a non trivial problem.

I dug deeper and came across Bag of Words (BOW) and Text Frequency - Inverse Document Frequency (TFIDF), which seemed suitable at first glance. But here I faced other problems: If I feed all the job titles to TFIDF, the resulting word-weight-vectors will probably be very large (in the tenth of thousands). The search term on the other hand will mostly consist of between 1 and 5 words (we currently match the job title only). Hence, the neural network must be able to reliably map an ultra sparse input vector to one of a few thousand basic jobs. This sounds very difficult for me and I doubt a good classification quality.

Another problem with BOW and TFIDF is, that they cannot handle typos and new words (I guess). They cannot be found in TFIDF's word list, which results in a vector filled with zeros. To sum it up: I was first excited to use TFIDF, but now think it doesn't work well for what I want to do.

Thinking more about it, I now have doubt if neural networks or other machine learning approaches are even good solutions for this task at all. Maybe there are much better algorithms in the field of natural language processing. This moment (before digging into NLP) I decided to first gather the opinions of some more experienced AI users, so I don't miss the best solution.

So what would be a useful approach to this in your opinion (best would be an approach that is capable of handling synonyms and typos)? Thanks in advance!

p. s.: I am currently thinking about feeding the whole job description into the TFIDF and also do matches for new incoming vacancies with the whole document (instead of job title only). This will expand the size of the word-weight-vector, but it will be less sparse. Does this seem logical to you?

Improve this question
$\endgroup$

4 Answers 4

Reset to default
1
$\begingroup$

This question has a number of parts to it.

First, you have a representation problem: what is the correct way to present textual data to your machine learning algorithm?

In this case, you chose to apply Bag-of-Words and then TFIDF scores. For English, this might be expected to produce on the order of 100,000 features, with each instance having only a few non-zero features.

If you want to go this route, you would typically also do some kind of feature selection to eliminate unimportant features from consideration. Depending on your task, you may be able to reduce the size of your input vectors quite dramatically while still getting good performance (for some tasks, to just 100 or so).

You're right that this might not be the most promising approach however.

My choice for this problem would be to use a compression classifier, like DMC. These have the advantage that they do not need any feature selection or pre-processing, and can easily handle new words or typos. They give state-of-the-art performance on tasks like spam-email classification.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for your advice so far. For your or anyones interest: Doing a test with around 60 000 samples I first got a feature size of around 31 000 from TfIdf (titles only). After applying some normalization to the inputs (e. g. lowercase transformation, removing stopwords, compound splitting) I was able to reduce the features to 13 000. I will have a look at DMC and its possible applications soon. $\endgroup$
    – Mr. Curious
    Aug 9, 2018 at 6:05
0
$\begingroup$

This is a huge growth area in the impact of AI on HR -- see all the companies we've found that do candidate matching for instance (disclaimer I work for CognitionX). Under the hood, there are techniques that don't rely on vocabulary such as Facebook's FastText but need more training data.

Here are some other resources Job matching using unsupervised learning (k-nearest neighbour) Excerpt from Oct 2018 paper -- using convolutional neural networks see paper

Improve this answer
$\endgroup$
0
$\begingroup$

Its definitely simpler task as NLP or mashine learning, its keyword based. And you have a little bit wrong view on that. See your example:

Example: java-developer, java web engineer and java software developer shall all be mapped to the profession java engineer. Not at all : java web and java are different jobs, while java software developer = java-developer and word software means nothing, cause java already stand for software. The info can NOT be mined from texts like job applications -> you have no link to a sence what title is , and better just create the mapping by hand - its not so long. Then, just look in text for key words and ignore other words

Improve this answer
$\endgroup$
1
  • $\begingroup$ Its definitely simpler task as NLP or mashine learning, its keyword based. Isn't a keyword based approach just a subpart of NLP? I mean, you will do something like word tokenizing, NER, maybe even POS analysis to improve performance. What's the difference to you? $\endgroup$
    – Mr. Curious
    Jun 4, 2019 at 14:19
0
$\begingroup$

I've been tackling a similar job title classification problem and used this paper as the basis for my approach: https://web.stanford.edu/~gavish/documents/phrase_based.pdf

Might find it useful.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .