4 votes

Top Frequent occurrence word effect in Model Efficiency?

Based on my experience, I did 2 tasks that is proven to improve the accuracy/score of my model. Normalization removing characters and symbols in a text lowercase folding Stopwords removal (as what ...
user avatar
4 votes
Accepted

Top Frequent occurrence word effect in Model Efficiency?

As far as I know, there are few aspects that would probably improve the model score: Normalization Lemmatization Stopwords removal (as you asked here) Based on your question, "is ...
user avatar
3 votes

How to go about classifying 1000 classes?

If you are asking for arbitrary ML task dealing with 1000+ classes the most straigtforward thing that comes to mind is the ImageNet - https://en.wikipedia.org/wiki/...
user avatar
3 votes
Accepted

How to use LSTM to generate a paragraph

Take the sentence that was generated by your LSTM and feed it back into the LSTM as input. Then the LSTM will generate the next sentence. So the LSTM is using it's previous output as it's input. That'...
user avatar
  • 182
3 votes

Is it possible to derive meaning from text by providing multiple ways of saying the same thing to a neural network?

No, it will not derive patterns of meaning, as the network has no understanding of language. What will happen, is that it picks up surface features (usually letter sequences) which are common between ...
user avatar
  • 5,062
2 votes

Does summing up word vectors destroy their meaning?

Summing up a sequence of word vector maybe used in practice sometimes. However, the operation of addition is non-reversible, meaning that once you sum up a few numbers, you cannot get the original ...
user avatar
  • 1,715
2 votes

What is the most accurate pretrained sentiment analysis model by 2019?

For best results, I'd recommend Google Cloud Machine Learning. It has [Natural Language Processing API] (https://cloud.google.com/natural-language/docs/basics) with Sentiment, Entity, and Entity-...
user avatar
2 votes
Accepted

Is a dataset of roughly 700 sentences of an average length of 15 words enough for text classification?

It depends on the number of classes; we are getting good results with about 40 training examples per class. A good way to get an idea about this is to run a test with an increasing set of training ...
user avatar
  • 5,062
2 votes

How to use LSTM to generate a paragraph

As you know, an LSTM language model takes in the past word and tries to predict the new one and continue over a loop. A sentence is divided into tokens and depending on different method, the tokens ...
user avatar
  • 1,715
2 votes

Is there an AI that can extract proper nouns from free text?

This is a hard problem, unless you have a list of proper nouns you want to recognise. If John Allen is in this list, then you can easily use a longest match to prefer it over John or Allen. The same ...
user avatar
  • 5,062
2 votes

Distinguishing text with opposite meanings in SVM (False Information Detection)

Going step by step: Preprocessing Preprocessing is a big deal in NLP, out there you'll find many tutorials describing the classic steps but few explanations about why and when you should actually ...
user avatar
1 vote

Which pre-processing steps are necessary for Deep Learning models to solve a document classification problem?

It depends on the type of model you use and the task, you are attempting to solve. Almost all the preprocessing steps that you mention remove some information from the text. If you think the removed ...
user avatar
  • 141
1 vote

Get the name of a merchant from records

The problem: You are facing a Natural Language problem called Named Entity Recognition (that's the key word you are looking for). But before you dive deep into it, have in mind it's best suited for ...
user avatar
1 vote

Training and Evaluating BERT and XLNET

You’ll want a reasonable GPU (probably 8GB+), but otherwise no special hardware needed.* You may need to tune down sequence length and batch size to fit your GPU; RAM will be the limiting factor. Don’...
user avatar
  • 266
1 vote
Accepted

Which algorithm can be used for extracting text patterns in tabular data?

You should first segregate the rejected samples. You can use then use string matching or something more complex (like creating embeddings and then, taking L2 distance between them) between the ...
user avatar
1 vote

Top Frequent occurrence word effect in Model Efficiency?

Based on my project, there is how i clean and doing some preparation on the data. Delete specific charaters ('\r', '\n', '"',) Change into the lowercase Delete some symbols Lemmatization (change ...
user avatar
1 vote
Accepted

Can I use one-hot vectors for text classification?

One hot encoding is a good strategy to apply with categorical variables that assume few possible values. The problem with text data is that you easily end up with corpora with a really large ...
user avatar
1 vote
Accepted

How do RNN's for sentiment classification deal with different sentence lengths?

One of the essential pre-processing we do on the corpus involves treating the variable-length sentences to a fixed length. There are various ways in which we can do this: Truncate This involves ...
user avatar
1 vote
Accepted

Text classification of non-equal length texts, should I pad left or right?

For any model that does not take a time series approach like an RNN does, the padding shouldn't make a difference. I prefer padding right simply because there also might be text you need to cut-off. ...
user avatar
  • 246
1 vote

NLP Identifying important key words in a corpus

I am sure there are complex methods to extract keywords, but the standard one which should serve as a strong baseline is the RAKE graph algorithm https://pypi.org/project/rake-nltk/. It should work ...
user avatar
1 vote

Top Frequent occurrence word effect in Model Efficiency?

The technical term for these words is "stop words". Have a look at Information Retrieval and indexing (eg TF/IDF) to make up your mind whether you want to remove them or not.
user avatar
  • 5,062
1 vote

Can artificial intelligence classify textual records?

AI can categorize documents very accurately. It is not a new application but in the last year the accuracy of the underlying algorithms such as text classifiers, and language models in general, has ...
user avatar
1 vote

When is it time to switch to deep neural networks from simple networks in text classification problems?

Source: https://blog.easysol.net/building-ai-applications/ When data is too big, complex and nonlinear it's time to try deep learning. It's always good to try to add some layers to see it can ...
user avatar
1 vote

How does the weight update formula for logistic regression work?

I think i found out how that works, so i made a short article about it . https://medium.com/@kourloskostas/python-spam-filter-86b21d7d1564 I hope it helps!
user avatar
  • 31
1 vote

Does summing up word vectors destroy their meaning?

But because the inputs have to have a fixed length Do they? Why? The go-to strategy would be to use an RNN (possibly with LSTM or GRUs, but probably not necessary) and train it to process input ...
user avatar
1 vote

How can a system recognize if two strings have the same or similar meaning?

Getting the intent of the sentence is not an easy task. To get you started on what to do, have a look on word vectors. You can also download pre-trained word2vec models. They help in getting ...
user avatar
1 vote

Is there any classifier that works best in general for NLP based projects?

First of all, there are multiple factors on how well models will work. Amount of data, source of data, hyperparameters, model type, training time etc... All of these will affect the accuracy. However, ...
user avatar
  • 1,715
1 vote

Is there any classifier that works best in general for NLP based projects?

The accuracy depends on various factors. Might not always be the algorithm. For example a cleaner data with a poor algorithm might still give better results and vice versa. What are the ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible