Questions tagged [data-preprocessing]

For questions related to the concept of data pre-processing, which includes, for example, cleaning, instance selection, normalization, transformation, feature extraction or selection.

Filter by
Sorted by
Tagged with
2
votes
0answers
7 views

How should I deal with multi-dimensional tensors for nodes in a graph convolution network?

How to work with GCN when the features of each node is not a 1D vector? For example, if the graph has N nodes and each node has features of the form $C \times D \times E$. Also, is there an open-...
2
votes
1answer
34 views

Do I need to denormalise results in linear regression?

I have learned so far how to linear regression with one or multiple features. So far, so good, everything seems to work fine, at least for my first simple examples. However, I now need to normalise ...
2
votes
0answers
20 views

Do I have to downsample the input and upsample the output of the neural network when implementing the NICE algorithm?

Consider that my input is an RGB image. The size of my image is $N\times N$. I'm trying to implement NICE algorithm presented by Dinh. The bijective function $f: \mathbb{R}^d \to \mathbb{R}^d$ maps $X$...
2
votes
0answers
16 views

What is the difference between training a model with RGB images and using only the color channels separately?

What is the difference between training a model with RGB images and using only the color channels separately (like only the red channel, green channel, etc.)? Would the model also learn patterns ...
1
vote
1answer
22 views

Is text preprocessing really all that necessary for NLP?

As a first step in many NLP courses, we learn about text preprocessing. The steps include lemmatization, removal of rare words, correcting typos etc. But I am not so sure about the actual ...
3
votes
1answer
34 views

How should I deal with variable-length inputs for neural networks?

I am a very beginner in the field of AI. I am basically a Pharma Professional without much coding experience. I use GUI based tools for the neural network. I am trying to develop an ANN that receives ...
0
votes
0answers
6 views

How to use a cv2 image augmentation function with tensorflow tf.data.Dataset? [migrated]

I am using tf.data.Dataset to create my dataset and training a CNN with keras. I need to apply masks on the images, and the mask depends on the shape of the image, ...
0
votes
0answers
21 views

How should I clean the columns of the dataset “Application-Layer DDoS Dataset” from Kaggle? [migrated]

I'm trying to get the right kind of data to build my model. I'm using TensorFlow's DNN classifier for this model. The dataset I'm using is Application-Layer DDoS Dataset from Kaggle. I've encoded the ...
2
votes
0answers
22 views

Can the addition of low-quality images to the training dataset increase the network performance?

I already trained a deep neural network called YOLO (You Only Look Once) with high-quality images (1920 by 1080 pixels) for a detection task. The result for mAP and IOU were 93% and 89% respectively. ...
0
votes
0answers
25 views

Image data generator for my (x, 12, 370, 235, 3) dataset [migrated]

I have made a model with 12 2D CNN inputs. I suddenly realized that the ImageDataGenerator from from keras.preprocessing import image don't accept inputs with ...
2
votes
1answer
34 views

How should I deal with variable input sizes for a neural network classifier?

I am currently working on a project, where I have a sensor in a shoe that records the $X, Y, Z$ axes, from an acceleration and gyroscope sensor. Every millisecond, I get 6 data points. Now, the goal ...
1
vote
1answer
25 views

Is it recommended to remove stop words before named entity recognition?

Removing stop words can significantly speed up named entity recognition (NER) modeling by reducing the number of tokens in a document. Are stop words critical to get correct NER performance?
1
vote
1answer
34 views

Using three image datasets with different image sizes to train a CNN

I've just started with AI and CNN networks. I have two NIFTI images dataset, one with (240, 240) dimensions and the other one with (256, 132). Each dataset is front a different hospital and machine. ...
2
votes
0answers
14 views

Language Learning feedback with AI

Is there a program under development that uses AI technology, like Siri, to "hold hands" so to speak with a language learner and coach them on accent, colloqiual expressions, or to let them guide the ...
0
votes
1answer
25 views

Choosing Data Augmentation smartly for different application

I'm trying to understand the role of data augmentation and how it can affect the performance/accuracy of a deep model. My target application is fire detection (on video frames), with almost 15K ...
0
votes
1answer
34 views

How to automatically detect and correct false information in columnar data?

I'm working on data cleaning and I'm stuck. I have a data set with 3 columns: id, age, and weight. Supposing I have an entry: ...
1
vote
0answers
15 views

Acoustic Input Data: Decibel or Pascals

In acoustics decibel levels were defined to solve an issue with showing values that are interpretive, understandable, and easy to communicate in contrast to intensity or pressure in Pascals. $dB = ...
1
vote
0answers
12 views

EEG and Accelerometer Neural Network

I have frequency EEG data from fall and non-fall events and I am trying to incorporate it with accelerometer data that was collected at the same time. One approach is, of course, to use two separate ...
0
votes
0answers
8 views

Hashing images detects false duplicates

Like I said in the title, I used a code I found to identify duplicate images however there are many false alarms. Can someone give me a link to some code I could use that really works? The problem ...
2
votes
0answers
86 views

Suggestion for finding the stable regions in spiral galaxy data?

I am working with a data set that consists of the actual pitch angle (given as PA(Y)) and the pitch angle at each radii (listed from 1 to 217). In the image below, ...
2
votes
1answer
83 views

How to define the “Pre-Processing” in machine learning?

Is every process1 that is done on the data before we train the model is always called the pre-processing part? Or are there some processes which are not included? 1"Every process" is includes data ...
1
vote
0answers
25 views

Are there free and easy-to-use annotation tools for 3D bounding boxes?

Are there free and easy-to-use annotation tools for 3D bounding boxes? There is a lot of annotation tools out there, but most of them only good for 2d boxes.
1
vote
0answers
16 views

learning object recognition of primitive shapes through transfer learning problem

Question on transfer learning object classification (MobileNet_v2 with 75% number of parameters) with my own synthetic data: I made my own dataset of three shapes: triangles, rectangles and spheres. ...
0
votes
0answers
53 views

Focal loss for imbalanced multi class classification in Pytorch

I want an example code for Focal loss in PyTorch for a model with three class prediction. My model outputs 3 probabilities. ...
1
vote
1answer
44 views

How can I merge two datasets? [closed]

I want to merge 2 data sets in one, but don't know the right approach to do it. The datasets are similar, the last column is the same - will or not them buy a product. In the first dataset, users who ...
5
votes
1answer
242 views

What is “conditioning” on a feature?

On page 98 of Jet Substructure at the Large Hadron Collider: A Review of Recent Advances in Theory and Machine Learning the author writes; Redacted phase space: Studying the distribution of inputs ...
0
votes
0answers
14 views

What's the correct approach to standardise data from a time-series used for LSTM neural network predictions?

This question discusses the same model mentioned in Why is the value range of my LSTM model's prediction different from my test labels? Repeating the main points: I am using LSTM to do time-...
0
votes
3answers
68 views

How can I remove the noise from an EEG signal?

I am working on a project that takes signals from the brain and preprocesses them and then makes the machine learn about what human is thinking about. But I am struck in preprocessing of signal (...
0
votes
0answers
11 views

Is there any reason to believe a ml pipeline that works on dataset A will work on dataset B where both have similar meta features?

I’m working on generating an automl pipeline(a combination of data cleaning and transformation algorithms applied to a dataset then generate a model) that works on a new dataset by looking for past ...
1
vote
1answer
45 views

How to rescale data to its original range after MinMaxScaler?

I'm using sklearn's MinMaxScaler in order to scale my data down. However, it would be nice to be able to rescale it back to its original range. Is there any way I ...
3
votes
1answer
54 views

Why do we normalize data in a deep neural network?

I have asked this question a number of times, but I always get confusing answers to this, like "normalized data works better", "data lives in the same scale" How can ...
1
vote
1answer
44 views

Do I need to use a pre-processed dataset to classify comments?

I want to use Machine Learning for text classification, more precisely, I want to determine whether a text (or comment) is positive or negative. I can download a dataset with 120 million comments. I ...
1
vote
3answers
47 views

Are there tools to help clean a large dataset so that it only contains faces?

I have a fairly large dataset consisting of different images with and without persons that I want to use for a project. The problem is that I only want the pictures that contain faces, and it is best ...
1
vote
2answers
57 views

What is the impact of scaling the features on the performance of the model?

I am trying to generate a model that uses several physicochemical properties of a molecule (including number of atoms, number of rings, volume, etc.) to predict a numeric value $Y$. I would like to ...
2
votes
2answers
88 views

Which of these two numerical methods for z-score normalisation is preferable, in multivariate linear regression?

In the exercise Exercise 3: Multivariate Linear Regression, by Andrew Ng, the author suggests to "scale both types of inputs by their standard deviations and set their means to zero". $$x_{n e w}=\...
1
vote
1answer
72 views

When should I use feature learning as opposed to feature engineering?

With the advancement of deep learning and a few others automated features learning techniques, manual feature engineering started becoming obsolete. Any suggestion on when to use manual feature ...
2
votes
0answers
144 views

Preprocessing of training dataset for machine learning

I'm developing a log analyzer to predict and find errors in an equipment. Each logged data contains the following format: ...