Tag Info

Hot answers tagged data-preprocessing

17 votes

How can I encode angle data to train neural networks?

The main problem with simply using the values $\alpha \in [0, 2\pi]$ is that semantically $0 = 2\pi$, but numerically $0$ and $2\pi$ are maximally far apart. A common way to encode this is by a vector ...
• 1,327
5 votes
Accepted

What is "conditioning" on a feature?

This is conditioning in the sense of conditional probability. The idea is that the authors have some "standard physically-inspired features". They are splitting the data up into bins based on the ...
• 9,047
4 votes
Accepted

Should I grey-scale the coloured frames/channels to build the approximation of the state?

We can't say for sure which approach would work best in the general case. If you have domain knowledge, you can make a better guess. You'll basically want to answer the question: which information is ...
• 9,804
4 votes

How should I deal with variable-length inputs for neural networks?

The most common way people deal with inputs of varying length is padding. You first define the desired sequence length, i.e. the input length you want your model to have. Then any sequences with a ...
• 3,153
4 votes
Accepted

How to solve the problem of too big activations when using genetic algorithms to train neural networks?

Your inputs should stay in a low range. Ideally for neural networks, the inputs are normalised to mean 0, standard deviation 1. I suspect this applies equally well to GA-driven NNs as gradient-driven ...
• 26.6k
4 votes
Accepted

Does the term "data augmentation" imply increasing the training dataset?

I'm not familiar with any "authoritative" single definition somewhere, or not sure who used the term first, but I would personally indeed agree with the reviewer you mention. In fact I've ...
• 9,804
3 votes

Why do we normalize data in a deep neural network?

I answered a similar question earlier and here is a piece of my answer that i think covers your question: Batch normalization's assistance to neural networks wasn't really understood for the longest ...
• 2,339
3 votes

Would this relatively small dataset be enough to train a CNN?

Your data set would be what is called "unbalanced' and this can lead to problems in developing an accurate classifier. The best thing to do (which you might not be able to do) is to find more ...
• 694
3 votes

Would this relatively small dataset be enough to train a CNN?

It is somewhat risky to discuss data independently with your learning mechanism. There is actually no such thing as good data or a good learner. There is only data that is good WITH a particular ...
• 7,385
3 votes
Accepted

Does data skew matter in classification problem?

Yes. Skewed data is one of the biggest problems in AI applications. As you rightly identified, the real world distribution is skewed. Doing a random sampling results in one major issue of an uneven ...
3 votes
Accepted

How does the RL agent understand motion if it gets only one image as input?

In the article Playing Atari with Deep Reinforcement Learning, Mnih et al, 2013, which was a major outbreak in Deep Reinforcement learning (especially in Deep Q learning), they don't feed only the ...
• 591
3 votes

How to perform prediction when some features have missing values?

You should look into "missing values". This is an entire research field in itself. First, you need to identify the type of missing values: They can be missing purely at random. Whether they ...
3 votes

Why do we resize images before using them for object detection?

There are different questions and even different lines of thought here. Let's go through them On resizing Why do we need to resize? To fit the network input which is fixed when nets are no Fully ...
• 1,068
2 votes
Accepted

Why is the short-time Fourier transform used for preprocessing audio samples?

Fourier transform is used to transform audio data to get more information (features). For example, raw audio data usually represented by a one-dimensional array, ...
• 2,631
2 votes
Accepted

How should I encode the input which are 5 cards from a deck of 52 cards?

Depends on how your game is played. Is there any meaning assigned to the order of cards, or are all 5 played simultaneously? If order matters, use 5 one-hot vectors so you can choose how to order them,...
• 933
2 votes

Do I need to use a pre-processed dataset to classify comments?

Here's a list of some of the best python libraries for natural language processing. Natural Language Toolkit (nltk) Covers all the basic functions and NLP tools ...
2 votes
Accepted

When should I use feature learning as opposed to feature engineering?

manual feature engineering started becoming obsolete That is wrong. Any suggestion on when to use manual feature engineering, feature learning or a combination of the two? Deep learning is ...
• 1,025
2 votes

Is text preprocessing really all that necessary for NLP?

It all depends on the quality of data. Due to old rule "Garbage in, garbage out" link , if you have bad quality data(data redundancy, unstructured data, too much memory, etc) your results won't be ...
• 143
2 votes
Accepted

How to rescale data to its original range after MinMaxScaler?

You can use the function inverse_transform of the created MinMaxScaler object. See also this Stack Overflow question for other ...
• 37.1k
2 votes

How should I deal with variable input sizes for a neural network classifier?

It is much simpler to process the data in a different way. Since you're using temporal data a common practice is to define a priori a minimum time-step, usually called $\textit{granularity}$, which ...
• 4,773
2 votes

Is it recommended to remove stop words before named entity recognition?

It depends how you recognise the entities. If you do a simple gazetteer lookup, then it could be faster, as you have fewer tokens to deal with. However, if you use contextual rules, then stop words ...
• 5,262
2 votes

How to automatically detect and correct false information in columnar data?

While I can see that there are some heuristics that can tell you whether an entry is 'weird', I don't see any way that you can correct this. Where would you get a correct value from? I would perhaps ...
• 5,262
2 votes
Accepted

How to define the "Pre-Processing" in machine learning?

Data preprocessing consists of all those techniques used to generate the final datasets (with an appropriate size, structure, and format) for the machine learning algorithms or models. Data ...
• 37.1k
2 votes

Are my steps correct for a proper classification of a sick brain?

It looks like everything you want is available with the Deep Learning Toolkit (DLTK) for Medical Imaging There is also a blog: An Introduction to Biomedical Image Analysis with TensorFlow and DLTK ...
• 1,698
2 votes
Accepted

Is it necessary to standardise the expected output

It depends, as mentioned in the comments, on your model and labels. For example, how would you use standardization on a multi-classification problem? Generally, standardization is more favorable for ...
• 313
2 votes
Accepted

When to convert data to word embeddings in NLP

If you have to move a lot of data around during training (like retrieving batches from disk/network/what have you), it's much faster to do so as a rank-3 tensor of [batches, documents, indices] than ...
• 436
2 votes
Accepted

How to mathematically describe the convolution operation (with a Gaussian kernel)?

Mathematically, the convolution is an operation that takes two functions, $f$ and $g$, and produces a third function, $h$. Concisely, we can denote the convolution operation as follows f \circledast ...
• 37.1k
2 votes
Accepted

How robust are deep networks to class imbalance?

@nbro pointed out the paper A systematic study of the class imbalance problem in convolutional neural networks, which tested class imbalance LeNet for MNIST, on a custom CNN for CIFAR-10, and on ...
• 1,400
2 votes
Accepted

In OCR, how should I deal with the warped text on the sides of oval objects?

There are many papers on this but the following is a good start: How to unwrap wine labels programmatically. The author includes source code in Python. You mentioned you do not want to do a ...
• 1,698
2 votes
Accepted

Does feature scaling have any benefits if all features are on the same scale?

If all you features are binary, then, you don't need to apply normalization on them. Since their values are on the same scale already.

Only top scored, non community-wiki answers of a minimum length are eligible