Questions tagged [data-preprocessing]

For questions related to the concept of data pre-processing, which includes, for example, cleaning, instance selection, normalization, transformation, feature extraction or selection.

Filter by
Sorted by
Tagged with
2
votes
1answer
45 views

How to define the “Pre-Processing” in machine learning?

Is every process1 that is done on the data before we train the model is always called the pre-processing part? Or are there some processes which are not included? 1"Every process" is includes data ...
0
votes
0answers
16 views

Problem in creating a Numpy Dataset for Classification problem

I am trying to train a classification Model to classify Images, but my images here are Numpy arrays of 1's and 0's as shown in the Image, For now, I have tried, ...
1
vote
0answers
19 views

Are there free and easy-to-use annotation tools for 3D bounding boxes?

Are there free and easy-to-use annotation tools for 3D bounding boxes? There is a lot of annotation tools out there, but most of them only good for 2d boxes.
1
vote
0answers
15 views

learning object recognition of primitive shapes through transfer learning problem

Question on transfer learning object classification (MobileNet_v2 with 75% number of parameters) with my own synthetic data: I made my own dataset of three shapes: triangles, rectangles and spheres. ...
0
votes
0answers
32 views

Focal loss for imbalanced multi class classification in Pytorch

I want an example code for Focal loss in PyTorch for a model with three class prediction. My model outputs 3 probabilities. ...
1
vote
1answer
44 views

How can I merge two datasets? [closed]

I want to merge 2 data sets in one, but don't know the right approach to do it. The datasets are similar, the last column is the same - will or not them buy a product. In the first dataset, users who ...
5
votes
1answer
205 views

What is “conditioning” on a feature?

On page 98 of Jet Substructure at the Large Hadron Collider: A Review of Recent Advances in Theory and Machine Learning the author writes; Redacted phase space: Studying the distribution of inputs ...
0
votes
0answers
14 views

What's the correct approach to standardise data from a time-series used for LSTM neural network predictions?

This question discusses the same model mentioned in Why is the value range of my LSTM model's prediction different from my test labels? Repeating the main points: I am using LSTM to do time-...
0
votes
2answers
52 views

How can I remove the noise from an EEG signal?

I am working on a project that takes signals from the brain and preprocesses them and then makes the machine learn about what human is thinking about. But I am struck in preprocessing of signal (...
0
votes
0answers
11 views

Is there any reason to believe a ml pipeline that works on dataset A will work on dataset B where both have similar meta features?

I’m working on generating an automl pipeline(a combination of data cleaning and transformation algorithms applied to a dataset then generate a model) that works on a new dataset by looking for past ...
1
vote
1answer
41 views

How to rescale data to its original range after MinMaxScaler?

I'm using sklearn's MinMaxScaler in order to scale my data down. However, it would be nice to be able to rescale it back to its original range. Is there any way I ...
3
votes
1answer
52 views

Why do we normalize data in a deep neural network?

I have asked this question a number of times, but I always get confusing answers to this, like "normalized data works better", "data lives in the same scale" How can ...
1
vote
1answer
42 views

Do I need to use a pre-processed dataset to classify comments?

I want to use Machine Learning for text classification, more precisely, I want to determine whether a text (or comment) is positive or negative. I can download a dataset with 120 million comments. I ...
1
vote
2answers
36 views

Are there tools to help clean a large dataset so that it only contains faces?

I have a fairly large dataset consisting of different images with and without persons that I want to use for a project. The problem is that I only want the pictures that contain faces, and it is best ...
1
vote
2answers
56 views

What is the impact of scaling the features on the performance of the model?

I am trying to generate a model that uses several physicochemical properties of a molecule (including number of atoms, number of rings, volume, etc.) to predict a numeric value $Y$. I would like to ...
2
votes
2answers
83 views

Which of these two numerical methods for z-score normalisation is preferable, in multivariate linear regression?

In the exercise Exercise 3: Multivariate Linear Regression, by Andrew Ng, the author suggests to "scale both types of inputs by their standard deviations and set their means to zero". $$x_{n e w}=\...
1
vote
1answer
70 views

When should I use feature learning as opposed to feature engineering?

With the advancement of deep learning and a few others automated features learning techniques, manual feature engineering started becoming obsolete. Any suggestion on when to use manual feature ...
2
votes
0answers
141 views

Preprocessing of training dataset for machine learning

I'm developing a log analyzer to predict and find errors in an equipment. Each logged data contains the following format: ...