first of all I want to specify the data available and what needs to be achieved: I have a huge amount of vacancies (in the millions). The information about the job title and the job description of each vacancy are stored separately. I also have a list of professions (around 3000), to which the vacancies shall be mapped.
Example: java-developer, java web engineer and java software developer shall all be mapped to the profession java engineer.
Now about my current researches and problems: Since a lot of potential training data is present, I thought a machine learning approach could be useful. I have been reading about different algorithms and wanted to give neural networks a shot.
Very fast I faced the problem, that I couldn't find a satisfying way to transform text of variable length to numerical vectors of constant size (needed by neural networks). As discussed here, this seems to be a non trivial problem.
I dug deeper and came across Bag of Words (BOW) and Text Frequency - Inverse Document Frequency (TFIDF), which seemed suitable at first glance. But here I faced other problems: If I feed all the job titles to TFIDF, the resulting word-weight-vectors will probably be very large (in the tenth of thousands). The search term on the other hand will mostly consist of between 1 and 5 words (we currently match the job title only). Hence, the neural network must be able to reliably map an ultra sparse input vector to one of a few thousand basic jobs. This sounds very difficult for me and I doubt a good classification quality.
Another problem with BOW and TFIDF is, that they cannot handle typos and new words (I guess). They cannot be found in TFIDF's word list, which results in a vector filled with zeros. To sum it up: I was first excited to use TFIDF, but now think it doesn't work well for what I want to do.
Thinking more about it, I now have doubt if neural networks or other machine learning approaches are even good solutions for this task at all. Maybe there are much better algorithms in the field of natural language processing. This moment (before digging into NLP) I decided to first gather the opinions of some more experienced AI users, so I don't miss the best solution.
So what would be a useful approach to this in your opinion (best would be an approach that is capable of handling synonyms and typos)? Thanks in advance!
p. s.: I am currently thinking about feeding the whole job description into the TFIDF and also do matches for new incoming vacancies with the whole document (instead of job title only). This will expand the size of the word-weight-vector, but it will be less sparse. Does this seem logical to you?