Questions tagged [data-preprocessing]

For questions related to the concept of data pre-processing, which includes, for example, cleaning, instance selection, normalization, transformation, feature extraction or selection.

25 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2
votes
0answers
16 views

How does sampling works in case of imbalanced image datasets?

I am solving a problem of image classification of the image dataset for 3 classes. Dataset is highly imbalanced. How will sampling (either over- or under-sampling) work in that case? Should I remove (...
2
votes
0answers
14 views

How to feed key-value features (aggregated data) to LSTM?

I have the following time-series aggregated input for an LSTM-based model: ...
2
votes
0answers
10 views

How should I deal with multi-dimensional tensors for nodes in a graph convolution network?

How to work with GCN when the features of each node is not a 1D vector? For example, if the graph has N nodes and each node has features of the form $C \times D \times E$. Also, is there an open-...
2
votes
0answers
26 views

Do I have to downsample the input and upsample the output of the neural network when implementing the NICE algorithm?

Consider that my input is an RGB image. The size of my image is $N\times N$. I'm trying to implement NICE algorithm presented by Dinh. The bijective function $f: \mathbb{R}^d \to \mathbb{R}^d$ maps $X$...
2
votes
0answers
16 views

What is the difference between training a model with RGB images and using only the color channels separately?

What is the difference between training a model with RGB images and using only the color channels separately (like only the red channel, green channel, etc.)? Would the model also learn patterns ...
2
votes
0answers
22 views

Can the addition of low-quality images to the training dataset increase the network performance?

I already trained a deep neural network called YOLO (You Only Look Once) with high-quality images (1920 by 1080 pixels) for a detection task. The result for mAP and IOU were 93% and 89% respectively. ...
2
votes
0answers
14 views

Language Learning feedback with AI

Is there a program under development that uses AI technology, like Siri, to "hold hands" so to speak with a language learner and coach them on accent, colloqiual expressions, or to let them guide the ...
2
votes
0answers
89 views

Suggestion for finding the stable regions in spiral galaxy data?

I am working with a data set that consists of the actual pitch angle (given as PA(Y)) and the pitch angle at each radii (listed from 1 to 217). In the image below, ...
2
votes
0answers
144 views

Preprocessing of training dataset for machine learning

I'm developing a log analyzer to predict and find errors in an equipment. Each logged data contains the following format: ...
1
vote
0answers
20 views

What pre-processing of the image is needed before feeding it into the convolutional neural network?

I can't figure out what preprocessing of the image is needed before feeding it into the convolutional neural network. For example, I want to recognize circles on a 1000 by 1000 px photo. The learning ...
1
vote
0answers
15 views

How can raw data from a motion sensor (like an IMU) reduced to the main points of the data

How can I reduce the caputured movement data of a person in a way, that I have filtered the main features of the movement. Or how can I detect pattern/ main features in that data set? I captured some ...
1
vote
0answers
18 views

Acoustic Input Data: Decibel or Pascals

In acoustics decibel levels were defined to solve an issue with showing values that are interpretive, understandable, and easy to communicate in contrast to intensity or pressure in Pascals. $dB = ...
1
vote
0answers
12 views

EEG and Accelerometer Neural Network

I have frequency EEG data from fall and non-fall events and I am trying to incorporate it with accelerometer data that was collected at the same time. One approach is, of course, to use two separate ...
1
vote
0answers
27 views

Are there free and easy-to-use annotation tools for 3D bounding boxes?

Are there free and easy-to-use annotation tools for 3D bounding boxes? There is a lot of annotation tools out there, but most of them only good for 2d boxes.
1
vote
0answers
16 views

learning object recognition of primitive shapes through transfer learning problem

Question on transfer learning object classification (MobileNet_v2 with 75% number of parameters) with my own synthetic data: I made my own dataset of three shapes: triangles, rectangles and spheres. ...
1
vote
0answers
56 views

Focal loss for imbalanced multi class classification in Pytorch

I want an example code for Focal loss in PyTorch for a model with three class prediction. My model outputs 3 probabilities. ...
1
vote
1answer
49 views

Is there an LSTM-based unsupervised learning algorithm to label a dataset of curves?

I have a big amount of light curves (image below). I am trying to label the points as signal or background (the signal appears usually periodically, several times, for a given light curve). More ...
0
votes
1answer
28 views

Can I find a mapping that minimizes the maximum distance ratio of certain vectors?

Let's say we have several vector points. My goal is to distinguish the vectors, so I want to make them far from each other. Some of them are already far from each other, but some of them can be ...
0
votes
0answers
18 views

Do we scale our target feature in regression problems?

I know this is a basic problem, but still could not find answer, I feel like most books/tutorials avoid talking about scaling the output feature instead they just mention scaling input features. So ...
0
votes
0answers
16 views

Are there any general guidelines for dealing with imbalanced data through upsampling or downsampling?

Are there any general guidelines for dealing with imbalanced data through upsampling/downsampling? This Google developer guide suggests performing downsampling with upweighting, but for the most ...
0
votes
0answers
48 views

How can I group the entries of the network traffic by their similarity?

I have the traffic of my network (with hundreds of entries). Below I am showing only 9 entries. ...
0
votes
0answers
8 views

Hashing images detects false duplicates

Like I said in the title, I used a code I found to identify duplicate images however there are many false alarms. Can someone give me a link to some code I could use that really works? The problem ...
0
votes
0answers
15 views

What's the correct approach to standardise data from a time-series used for LSTM neural network predictions?

This question discusses the same model mentioned in Why is the value range of my LSTM model's prediction different from my test labels? Repeating the main points: I am using LSTM to do time-...
0
votes
3answers
77 views

How can I remove the noise from an EEG signal?

I am working on a project that takes signals from the brain and preprocesses them and then makes the machine learn about what human is thinking about. But I am struck in preprocessing of signal (...
0
votes
0answers
11 views

Is there any reason to believe a ml pipeline that works on dataset A will work on dataset B where both have similar meta features?

I’m working on generating an automl pipeline(a combination of data cleaning and transformation algorithms applied to a dataset then generate a model) that works on a new dataset by looking for past ...