2
$\begingroup$

I am working on an Intent detection problem for a chatbot in Java. So I need to convert words from String to a double[] format. I tried using wordToVec(deeplearning4j), but it does not return a vector for words not present in the training data.

e.g. My dataset for wordToVec.train() does not contain the word "morning". So wordToVec.getWordVector("morning") returns a null value.

There is no need to find the coorelation between two words(like in word2vec), but it should be able to give me some sort of vector representation for any word.

Here are some things I thought of-

  1. I could use a fixed length hash function and convert resulatant hash into vector.(Will Hash Collision be strong enough to be an issue in this case?)
  2. I could initialize for each word a vector of huge length as zero, and set its elements as the ASCII value-64. e.g. Keeping Maximum vector length as 10, AND would be represented as- [1,14,4,0,0,0,0,0,0,0], and normalize this. Is there a better solution to this problem?

Here is the code I used to train the model-

public static void trainModel() throws IOException
    {
        //These lines simply generate the dataset into a format readable by wordToVec
        utilities.GenRawSentences.genRaw();

        dataLocalPath = "./TrainingData/";
        String filePath = new File(dataLocalPath, "raw_sentences.txt").getAbsolutePath();
        //Data Generation ends   

        SentenceIterator iter = new BasicLineIterator(filePath);
        TokenizerFactory t = new DefaultTokenizerFactory();
        t.setTokenPreProcessor(new CommonPreprocessor());

        VocabCache<VocabWord> cache = new AbstractCache<>();
        WeightLookupTable<VocabWord> table = new InMemoryLookupTable.Builder<VocabWord>()
                .vectorLength(64)
                .useAdaGrad(false)
                .cache(cache).build();

        Word2Vec vec = new Word2Vec.Builder()
                .minWordFrequency(1)
                .iterations(5)
                .epochs(1)
                .layerSize(64)
                .seed(42)
                .windowSize(5)
                .iterate(iter)
                .tokenizerFactory(t)
                .lookupTable(table)
                .vocabCache(cache)
                .build();

        vec.fit();

        //Saves the model for use in other programs
        WordVectorSerializer.writeWord2VecModel(vec, "./Models/WordToVecModel.txt");

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.