We changed our privacy policy. Read more.

Questions tagged [data-preprocessing]

For questions related to the concept of data pre-processing, which includes, for example, cleaning, instance selection, normalization, transformation, feature extraction or selection.

Filter by
Sorted by
Tagged with
6
votes
1answer
140 views

How to solve the problem of too big activations when using genetic algorithms to train neural networks?

I am trying to create a fixed-topology MLP from scratch (with C#), which can solve some simple problems, such as the XOR problem and MNIST classification. The network will be trained purely with ...
5
votes
1answer
504 views

What is "conditioning" on a feature?

On page 98 of Jet Substructure at the Large Hadron Collider: A Review of Recent Advances in Theory and Machine Learning the author writes; Redacted phase space: Studying the distribution of inputs ...
4
votes
3answers
193 views

Would this relatively small dataset be enough to train a CNN?

Scenario: I am trying to create a dataset with images of choice for different animal classes. I am going to train those images for classification using CNN. Problem: Let's assume I somehow don't have ...
4
votes
1answer
222 views

How should I deal with variable-length inputs for neural networks?

I am a very beginner in the field of AI. I am basically a Pharma Professional without much coding experience. I use GUI based tools for the neural network. I am trying to develop an ANN that receives ...
4
votes
1answer
334 views

How should I deal with variable input sizes for a neural network classifier?

I am currently working on a project, where I have a sensor in a shoe that records the $X, Y, Z$ axes, from an acceleration and gyroscope sensor. Every millisecond, I get 6 data points. Now, the goal ...
4
votes
1answer
102 views

How to fill missing values in a dataset where some properties can be inputs and outputs?

I have a dataset with missing values, I would like to use machine learning methods to fill. In more detail, there are $n$ individuals, for which up to 10 properties are provided, all numerical. The ...
3
votes
1answer
152 views

Should I grey-scale the coloured frames/channels to build the approximation of the state?

I'm doing reinforcement learning, and I have a visual observation that I will use to build an input state for my agent. In the DeepMind's Atari paper, they greyscale the input image before they input ...
3
votes
1answer
78 views

How to perform prediction when some features have missing values?

Sorry if this is too noob question, I'm just a beginner. I have a data set with companies' info. There are 2 kinds of features: financial (revenue and so on) and general info (like the number of ...
3
votes
3answers
81 views

When to convert data to word embeddings in NLP

When training a network using word embeddings, it is standard to add an embedding layer to first convert the input vector to the embeddings. However, assuming the embeddings are pre-trained and frozen,...
3
votes
1answer
64 views

Why do we normalize data in a deep neural network?

I have asked this question a number of times, but I always get confusing answers to this, like "normalized data works better", "data lives in the same scale" How can ...
3
votes
2answers
50 views

Is pre-processing used in deep learning?

I'm new to deep learning. I wanted to know: do we use pre-processing in deep learning? Or it is only used in machine learning. I searched for it and its methods on the internet, but I didn't find a ...
2
votes
1answer
33 views

How to handle class imbalancing when the actual data are that way

My supervised learning training data are obtained from actual data; and in real cases, there's one class which happens less often than other classes, just around 5% of all cases. To be precise, the ...
2
votes
1answer
128 views

In OCR, how should I deal with the warped text on the sides of oval objects?

Consider an image that contains one can (or bottle, or any similar oval object), which has texts all over it. In the image below, I have many bottles, but you can assume that each image only contains ...
2
votes
1answer
81 views

How robust are deep networks to class imbalance?

Before deep learning, I worked with machine learning problems where the data had a large class imbalance (30:1 or worse ratios). At that time, all the classifiers struggled, even after under-sampling ...
2
votes
1answer
47 views

Are my steps correct for a proper classification of a sick brain?

I have a dataset with MRI of patients with a specific disease that affects the brain and another dataset with MRI of healthy patients. I want to create a classifier (using neural networks) to ...
2
votes
1answer
108 views

How to define the "Pre-Processing" in machine learning?

Is every process (such as data acquisition, splitting the data for validation, data cleaning, or feature engineering) that is done on the data before we train the model always called the pre-...
2
votes
2answers
63 views

Is feature engineer an important step for a deep learning approach?

I'd like to ask you if feature engineering is an important step for a deep learning approach. By feature engineering I mean some advanced preprocessing steps, such as looking at histogram ...
2
votes
1answer
45 views

Is data leakage relevant when scaling across samples?

I have a question about data leakage when pre-processing data for a neural network and whether data leakage actually applies in my instance. I have variance stabilising transformed genomic data. ...
2
votes
1answer
50 views

Should I remove the text overlaying some images in the dataset before training the CNN?

If I am attempting to train a CNN on some image data to perform image classification, but some of the images have pieces of text overlaying them (for the purpose of description to humans), then is it ...
2
votes
2answers
41 views

How to take the optimal batch_size for training a model?

I have an image dataset, which is composed of 113695 images for training and 28424 images for validation. Now, when I use ImageDataGenerator and ...
2
votes
2answers
75 views

How to train a neural network with a data set that in which the target is a mix of 0-1 label and numeric real value label?

I am running into an issue in which the the target (label collums) of my dataset contain a mixture of binary label (yes/no) and some numeric value label. The value of these numeric value (resource 1 ...
2
votes
2answers
92 views

Which of these two numerical methods for z-score normalisation is preferable, in multivariate linear regression?

In the exercise Exercise 3: Multivariate Linear Regression, by Andrew Ng, the author suggests to "scale both types of inputs by their standard deviations and set their means to zero". $$x_{n e w}=\...
2
votes
1answer
141 views

How should I encode the input which are 5 cards from a deck of 52 cards?

How should I design my input layer for the following classification problem? Input: 5 cards (from a deck of 52 cards) in a card game; Output: some classification using a neural network How should I ...
2
votes
1answer
2k views

How to label edited images after data augmentation?

I am new to neural networks, I've only started studying and learning about the subject a year ago, and I just started building my first neural network. The project is a little bit ambitious: A browser ...
2
votes
0answers
39 views

How to deal with a variable number of channels of the inputs?

I have a problem in which my input data may have a varying number of channels. Let me explain with an example. Imagine we have a classification problem in which we wish to identify if certain species ...
2
votes
0answers
61 views

Statistical method for selecting features for classification

I'm working on a classifier for the famous MNIST handwritten data set. I want to create a few features on my own, and I want to be able to estimate which feature might perform better before actually ...
2
votes
0answers
26 views

How does sampling works in case of imbalanced image datasets?

I am solving a problem of image classification of the image dataset for 3 classes. Dataset is highly imbalanced. How will sampling (either over- or under-sampling) work in that case? Should I remove (...
2
votes
0answers
28 views

How to feed key-value features (aggregated data) to LSTM?

I have the following time-series aggregated input for an LSTM-based model: ...
2
votes
0answers
19 views

How should I deal with multi-dimensional tensors for nodes in a graph convolution network?

How to work with GCN when the features of each node is not a 1D vector? For example, if the graph has N nodes and each node has features of the form $C \times D \times E$. Also, is there an open-...
2
votes
1answer
73 views

Do I need to denormalise results in linear regression?

I have learned so far how to linear regression with one or multiple features. So far, so good, everything seems to work fine, at least for my first simple examples. However, I now need to normalise ...
2
votes
0answers
64 views

Do I have to downsample the input and upsample the output of the neural network when implementing the NICE algorithm?

Consider that my input is an RGB image. The size of my image is $N\times N$. I'm trying to implement NICE algorithm presented by Dinh. The bijective function $f: \mathbb{R}^d \to \mathbb{R}^d$ maps $X$...
2
votes
0answers
26 views

What is the difference between training a model with RGB images and using only the color channels separately?

What is the difference between training a model with RGB images and using only the color channels separately (like only the red channel, green channel, etc.)? Would the model also learn patterns ...
2
votes
0answers
32 views

Can the addition of low-quality images to the training dataset increase the network performance?

I already trained a deep neural network called YOLO (You Only Look Once) with high-quality images (1920 by 1080 pixels) for a detection task. The result for mAP and IOU were 93% and 89% respectively. ...
2
votes
0answers
19 views

Language Learning feedback with AI

Is there a program under development that uses AI technology, like Siri, to "hold hands" so to speak with a language learner and coach them on accent, colloqiual expressions, or to let them guide the ...
2
votes
0answers
96 views

Suggestion for finding the stable regions in spiral galaxy data?

I am working with a data set that consists of the actual pitch angle (given as PA(Y)) and the pitch angle at each radii (listed from 1 to 217). In the image below, ...
2
votes
1answer
97 views

Validation Loss Fluctuates then Decrease alongside Validation Accuracy Increases

I was working on CNN. I modified the training procedure on runtime. As we can see from the validation loss and validation accuracy, the yellow curve does not fluctuate much. The green curve and red ...
2
votes
0answers
151 views

Preprocessing of training dataset for machine learning

I'm developing a log analyzer to predict and find errors in an equipment. Each logged data contains the following format: ...
2
votes
5answers
80 views

How do I increase the size of an (almost) balanced dataset?

I am trying to add more data points in my (almost) balanced dataset for training my neural network. I have come across techniques such as SMOTE or Random Over Sampling, but they work best for ...
1
vote
3answers
56 views

Are there tools to help clean a large dataset so that it only contains faces?

I have a fairly large dataset consisting of different images with and without persons that I want to use for a project. The problem is that I only want the pictures that contain faces, and it is best ...
1
vote
1answer
27 views

How can I address missing values for LSTM?

I'm a student and writing my first paper for submission on conference. I have a question there is a dataset below. this is temporal-spatial dataset. ...
1
vote
1answer
27 views

Does feature scaling have any benefits if all features are on the same scale?

By scaling features, we can prevent one feature from dominating the decisions of a model. For example, say heights (cm), and age (years) are two features in my data. Since range of heights is larger ...
1
vote
1answer
167 views

Why do we resize images before using them for object detection?

In object detection, we can resize images by keeping the ratio the same as the original image, which is often known as "letterbox" resize. My questions are Why do we need to resize images? ...
1
vote
1answer
124 views

How to mathematically describe the convolution operation (with a Gaussian kernel)?

I have to build a model where I pre-process the data with a Gaussian kernel. The data are an $n\times n$ matrix (i.e one channel), but not an image, thus I can't refer to this matrix as an image and ...
1
vote
2answers
77 views

Is text preprocessing really all that necessary for NLP?

As a first step in many NLP courses, we learn about text preprocessing. The steps include lemmatization, removal of rare words, correcting typos etc. But I am not so sure about the actual ...
1
vote
1answer
148 views

How to rescale data to its original range after MinMaxScaler? [closed]

I'm using sklearn's MinMaxScaler in order to scale my data down. However, it would be nice to be able to rescale it back to its original range. Is there any way I ...
1
vote
1answer
52 views

Do I need to use a pre-processed dataset to classify comments?

I want to use Machine Learning for text classification, more precisely, I want to determine whether a text (or comment) is positive or negative. I can download a dataset with 120 million comments. I ...
1
vote
1answer
87 views

When should I use feature learning as opposed to feature engineering?

With the advancement of deep learning and a few others automated features learning techniques, manual feature engineering started becoming obsolete. Any suggestion on when to use manual feature ...
1
vote
1answer
19 views

What to do when you have massive amount of data but you don't have enough computation power for training a machine learning model?

For example, I have a massive amount of data, but I have limited computational resources and time to train on the full data. Other cases may include, I have huge amounts of 360-degree images, where I ...
1
vote
1answer
46 views

Rescaling time-series data with very spiky pattern for training data in LSTM network

I am working with some time-series hydrology data. Our goal is to forecast the time series forward, meaning predicting the data 1 month, 3 months ,6 months into the future. The data itself(image below)...
1
vote
1answer
50 views

How to normalize images before training?

I have seen people normalize images by just dividing 255. But why? Why not use mean normalization or Z-score Normalization? I also came across this StackOverflow topic while searching but the answers ...