4

Based on my experience, I did 2 tasks that is proven to improve the accuracy/score of my model. Normalization removing characters and symbols in a text lowercase folding Stopwords removal (as what you asked) These process helped me improve my model since stopwords gave my model noise as I am using word frequency count to represent text. So based on ...


4

As far as I know, there are few aspects that would probably improve the model score: Normalization Lemmatization Stopwords removal (as you asked here) Based on your question, "is removing top frequent words (stopwords) will improve the model score?". The answer is, it depends on what kind of stopwords are you removing. The problem here is that ...


3

Take the sentence that was generated by your LSTM and feed it back into the LSTM as input. Then the LSTM will generate the next sentence. So the LSTM is using it's previous output as it's input. That's what makes it recursive. The intial word is just your base case. Also you should consider using GPT2 by open AI to do this. It's pretty impressive. https://...


3

No, it will not derive patterns of meaning, as the network has no understanding of language. What will happen, is that it picks up surface features (usually letter sequences) which are common between sentences with the same (or a similar) meaning. This approach is often used in chatbots for intent recognition. Sometimes it picks up subtle patterns that ...


2

As you know, an LSTM language model takes in the past word and tries to predict the new one and continue over a loop. A sentence is divided into tokens and depending on different method, the tokens are divided differently. Some model maybe character based models which simply uses each character as input and output. In this case you can treat punctuation as ...


2

Summing up a sequence of word vector maybe used in practice sometimes. However, the operation of addition is non-reversible, meaning that once you sum up a few numbers, you cannot get the original numbers back. However summing up a sequence of word vectors may work depending on your task. You should also normalize the values, or just use average value. For ...


2

For best results, I'd recommend Google Cloud Machine Learning. It has [Natural Language Processing API] (https://cloud.google.com/natural-language/docs/basics) with Sentiment, Entity, and Entity-Sentiment analysis. You can implement in C++, PHP, Python, or other languages. This does require running a virtual machine instance on Google Cloud. TensorFlow can ...


2

It depends on the number of classes; we are getting good results with about 40 training examples per class. A good way to get an idea about this is to run a test with an increasing set of training data, evaluating the result as you go along. Obviously, with a small set (eg 3 sentences per class), it will be very poor, but the accuracy should quickly ...


1

One hot encoding is a good strategy to apply with categorical variables that assume few possible values. The problem with text data is that you easily end up with corpora with a really large vocabulary. If I remember correctly the IMDb dataset contains around 130.000 unique words, which means that you should create a network with an input matrix of size 130....


1

For any model that does not take a time series approach like an RNN does, the padding shouldn't make a difference. I prefer padding right simply because there also might be text you need to cut-off. Then padding is more intuitive as you either cut-off a text if it's too long or pad a text when it's too short. Either way, when a model is trained a certain way,...


1

I am sure there are complex methods to extract keywords, but the standard one which should serve as a strong baseline is the RAKE graph algorithm https://pypi.org/project/rake-nltk/. It should work reasonably well in most text domains.


1

Based on my project, there is how i clean and doing some preparation on the data. Delete specific charaters ('\r', '\n', '"',) Change into the lowercase Delete some symbols Lemmatization (change base word with wordnet) Delete stopwords. With these following step, i get some improvement accuracy score on my model. My project: https://github.com/...


1

The technical term for these words is "stop words". Have a look at Information Retrieval and indexing (eg TF/IDF) to make up your mind whether you want to remove them or not.


1

AI can categorize documents very accurately. It is not a new application but in the last year the accuracy of the underlying algorithms such as text classifiers, and language models in general, has significantly improved. There are applications of language models which now surpass human performance. Microsoft is one of the leaders in this area. For ...


1

Source: https://blog.easysol.net/building-ai-applications/ When data is too big, complex and nonlinear it's time to try deep learning. It's always good to try to add some layers to see it can eliminate bias and don't lead to high variance. Deep learning models can be tweaked(hyperparameters) and regularized(parameters), and it is worth the work.


1

I think i found out how that works, so i made a short article about it . https://medium.com/@kourloskostas/python-spam-filter-86b21d7d1564 I hope it helps!


1

But because the inputs have to have a fixed length Do they? Why? The go-to strategy would be to use an RNN (possibly with LSTM or GRUs, but probably not necessary) and train it to process input sequentially and output the final classification of the paragraph. This has the advantage of being able to take into account word order and constellations, as ...


1

Getting the intent of the sentence is not an easy task. To get you started on what to do, have a look on word vectors. You can also download pre-trained word2vec models. They help in getting similarity of words and reasoning with words. To get the intent of a sentence, you can use LSTM. Fun fact most NLP algorithms strip away punctuation with is sufficient ...


1

First of all, there are multiple factors on how well models will work. Amount of data, source of data, hyperparameters, model type, training time etc... All of these will affect the accuracy. However, no classifier will work best in general. It all depends on the different factors, and not one can satisfy all, at least for now. For improving the accuracy, ...


1

The accuracy depends on various factors. Might not always be the algorithm. For example a cleaner data with a poor algorithm might still give better results and vice versa. What are the preprocessing techniques you are using? This preprocessing techniques article is a good starting point for html data. And by vectorising I assume you mean word2vec, use a ...


Only top voted, non community-wiki answers of a minimum length are eligible