1
$\begingroup$

I want to use Machine Learning for text classification, more precisely, I want to determine whether a text (or comment) is positive or negative. I can download a dataset with 120 million comments. I read the TensorFlow tutorial and they also have a text dataset. This dataset is already pre-processed, like the words are converted to integers and the most used words are in the top 10000.

Do I also have to use a pre-processed dataset like them? If yes, does it have to be like the dataset from TensorFlow? And which pages could help me to implement that kind of program?

My steps would be:

  1. find datasets
  2. preprocess them if needed
  3. feed them in the neural network
Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Here's a list of some of the best python libraries for natural language processing.

  • Natural Language Toolkit (nltk) Covers all the basic functions and NLP tools such as tokenization etc.

  • TextBolb This is a good library of beginners, it provides the nltk toolkit in a simplified format.

  • Spacy It is an advanced library and can be used in production code.

You can preprocess textual data in a number of ways. It depends on the type of task at hand and the size of the data.

From your question, I think you are referring to converting the words to a vector form word2vec. Here is a massive word2vect list from Google.

Also have a look at preprocessing techniques in NLP such as tf-idf etc.

Improve this answer
$\endgroup$
0

You must log in to answer this question.