I've lot of training data points (i.e in millions) and I've around few features but the issue with that is all the features are categorical data with 1 million+ categories in each.

So, I couldn't use one hot encoding because it's not efficient so I went with the other option which is embedding of fixed length. I've just used neural nets to compute embedding.

My question is can we use advanced NLP models like bert to extract embeddings for categorical data from my corpus? Is it possible? I've only asked it because I've only heard that bert is good for sentence embeddings.

Thank you.

  • $\begingroup$ Are your categorical features words/sentences? Why do you want to embed categorical features? $\endgroup$
    – Sharan
    Jan 20 '20 at 6:20
  • $\begingroup$ @Sharan It's just words not sentences. There are lot of categories and if I choose to use one hot encoding it would result in a extremely sparse data-set. So only other option is to use embeddings. $\endgroup$
    – user_12
    Jan 20 '20 at 19:06
  • $\begingroup$ Yes I am not suggesting one hot. I just wanted to know how you arrived at embeddings. BERT is very computationally expensive and it would be inefficient to use it for "1 million+" categories. I d suggest you to start off with something simple like Word2Vec. Also simple mathematical features might help in some case - analyticsvidhya.com/blog/2018/02/… $\endgroup$
    – Sharan
    Jan 21 '20 at 13:05
  • $\begingroup$ @Sharan I understand but I figured bert can understand the context better than any simple methods like word2vec etc.. So was just wondering if it could work on categories? $\endgroup$
    – user_12
    Jan 21 '20 at 13:48
  • $\begingroup$ This question needs more context. What exactly is the problem you are trying to solve? What are training data, what are features, what are the categories? Please clarify. $\endgroup$ Jan 28 '20 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.