3
$\begingroup$

I am trying to predict the number of likes an article or a post will get using a NN.

I have a dataframe with ~70,000 rows and 2 columns: "text" (predictor - strings of text) and "likes" (target - continuous int variable). I've been reading on the approaches that are taken in NLP problems, but I feel somewhat lost as to what the input for the NN should look like.

Here is what I did so far:

  1. Text cleaning: removing html tags, stop words, punctuation, etc...
  2. Lower-casing the text column
  3. Tokenization
  4. Lemmatization
  5. Stemming

I assigned the results to a new column, so now I have "clean_text" column with all the above applied to it. However, I'm not sure how to proceed.

In most NLP problems, I have noticed that people use word embeddings, but from what I have understood, it's a method used when attempting to predict the next word in a text. Learning word embeddings creates vectors for words that are similar to each other syntax-wise, and I fail to see how that can be used to derive the weight/impact of each word on the target variable in my case.

In addition, when I tried to generate a word embedding model using the Gensim library, it resulted in more than 50k words, which I think will make it too difficult or even impossible to onehot encode. Even then, I will have to one hot encode each row and then create a padding for all the rows to be of similar length to feed the NN model, but the length of each row in the new column I created "clean_text" varies significantly, so it will result in very big onehot encoded matrices that are kind of redundant.

Am I approaching this completely wrong? and what should I do?

$\endgroup$
3
$\begingroup$

I'll answer in a couple of stages.

I feel somewhat lost as to what the input for the NN should look like.

Your choices boil down to two options, each with their own multitude of variants:

  1. Vector Representation: Your input is a vector of the same size as your vocabulary where the elements represent the tokens in the input example. The most basic version of this is a bag-of-words (BOW) encoding with a 1 for each word that occurs in the input example and a 0 otherwise. Some other variants are (normalized) word counts or TF-IDF values. With this representation padding will not be necessary as each example will be encoded as a vector of the same size as the vocabulary. However, it suffers from a variety of issues: the input is high-dimensional and very sparse making learning difficult (as you note), it does not encode word order, and the individual word representations have little (TF-IDF) to no (BOW, counts) semantic information. It also limits your NN architecture to a feed-forward network, as more "interesting" architectures such a RNNs, CNNs, and transformers assume a matrix-like input, described below.

  2. Matrix Representation: Here your input representation is a matrix with each row being a vector (i.e. embedding) representation of the token at that index in the input example. How you actually get the pretrained embeddings into the model depends on a number of implementation-specific factors, but this stackoverflow question shows how to load embeddings from gensim into PyTorch. Here padding is necessary because the input examples will have variable numbers of tokens. This stackoverflow answer shows how to add zero padding in PyTorch. This representation will be significantly better than the vector representation as it is relatively low-dimensional and non-sparse, it maintains word order, and using pretrained word-embeddings means your model will have access to semantic information. In fact, this last point leads to your next question.

Learning word embeddings creates vectors for words that are similar to each other syntax-wise, and I fail to see how that can be used to derive the weight/impact of each word on the target variable in my case.

Word embeddings are based on the assumptions of distributional semantics, the core tenet of which is often quoted as "a word is characterized by the company it keeps". That is, the meaning of a word is how it relates to other words. In the context of NLP, models can make better decisions because similar words are treated similarly from the get-go.

For example, say that articles about furry pets get a lot of likes (entirely plausible if you ask me). However, the mentions of furry pets in these articles will be varied, including words like "dog", "cat", "chinchilla", "poodle", "doggo", "good boy", etc. An input representation that treats these mentions as completely distinct (such as BOW) will need to learn individual correlations between each word and the number of likes (that's a lot of learning). A well-trained word embedding, on the other hand, will be able to immediately group these mentions together and learn general correlations between groups of similar words and likes. Fair warning, this is a very imprecise description of why word embeddings work, but I hope it gives you some intuitive understanding.

Finally, since you're doing regression, make sure you choose your objective function accordingly. Mean squared error would be my first try.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for the detailed answer, this helps a lot $\endgroup$ – Najati Al-imam Aug 6 at 9:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.