Questions tagged [data-preprocessing]

For questions related to the concept of data pre-processing, which includes, for example, cleaning, instance selection, normalization, transformation, feature extraction or selection.

Filter by
Sorted by
Tagged with
1
vote
0answers
9 views

Multiple Inertia sensors system based for gestures recognition

I am a newbie to Machine Learning field as I am engaging to a personal project that I am trying to use the 6 degree of freedom Inertial Measurement Units(IMUs) measuring the Acceleration acting on 3 ...
2
votes
1answer
36 views

Should I remove the text overlaying some images in the dataset before training the CNN?

If I am attempting to train a CNN on some image data to perform image classification, but some of the images have pieces of text overlaying them (for the purpose of description to humans), then is it ...
0
votes
1answer
21 views

Can a neural network be trained on a dataset containing only values for true output for a classification problem?

I am using a dataset from Google which contains 1,27,000 data points on simulated concentrations of the atmosphere of exoplanets which can sustain life. So, the output label of all these data points ...
2
votes
1answer
56 views

How to perform prediction when some features have missing values?

Sorry if this is too noob question, I'm just a beginner. I have a data set with companies' info. There are 2 kinds of features: financial (revenue and so on) and general info (like the number of ...
1
vote
0answers
44 views

How much should we augment our training data?

I am wondering how much I should extend my training set with data augmentation. Is there somewhere a pre-defined number I can go with? Suppose I have 10000 images, can I go as far as 10x or 20x times, ...
0
votes
1answer
35 views

Do the rows of the design matrix refer to the observations or predictors?

I attempt to understand the formulation of dictionary learning for this paper: Depression Detection via Harvesting Social Media: A Multimodal Dictionary Learning Solution Multimodal Task-Driven ...
2
votes
3answers
57 views

When to convert data to word embeddings in NLP

When training a network using word embeddings, it is standard to add an embedding layer to first convert the input vector to the embeddings. However, assuming the embeddings are pre-trained and frozen,...
2
votes
2answers
28 views

How to take the optimal batch_size for training a model?

I have an image dataset, which is composed of 113695 images for training and 28424 images for validation. Now, when I use ImageDataGenerator and ...
1
vote
0answers
20 views

Can I resize my images after labeling them?

Is it okay if I label my images with their original size and then resize them, or should I first resize them and then label them? I mean do I need to recalibrate my labels if I resized my images?
0
votes
0answers
57 views

Multilabel stratified split for images/object detection

I am working on an object detection model and have thought of looking into stratified splits for the dataset. Now since I am doing object detection I have a variable number of "labels" for ...
0
votes
1answer
61 views

Is it necessary to standardise the expected output

Normalisation transform data into a range: $$X_i = \dfrac{X_i - Min}{Max-Min}$$ Practically, I found out that the model doesn't generalise well when using normalisation of input data, instead of ...
1
vote
1answer
43 views

Do I need to rotate the masks, if I also rotate the images and the masks are generated from the input?

I am training a neural network that takes an input (H, W, 3) and has the output of size (H', W', C). Now, to augment my dataset, ...
1
vote
0answers
18 views

How is the performance of a CNN trained with monochrome images on image recognition tasks degraded?

For CNN image recognition tasks, like object recognition/face recognition/object segmentation/posture recognition, are there experiment results about how much will the performance be degraded with ...
1
vote
0answers
30 views

How can I predict the label given a partial feature vector?

Most of the traditional machine learning algorithms need a feature vector of a constant dimension to predict the label. Which algorithms can be used to predict a class label with a shorter or ...
1
vote
2answers
62 views

How to train a neural network with a data set that in which the target is a mix of 0-1 label and numeric real value label?

I am running into an issue in which the the target (label collums) of my dataset contain a mixture of binary label (yes/no) and some numeric value label. The value of these numeric value (resource 1 ...
1
vote
1answer
40 views

Can I find a mapping that minimizes the maximum distance ratio of certain vectors?

Let's say we have several vector points. My goal is to distinguish the vectors, so I want to make them far from each other. Some of them are already far from each other, but some of them can be ...
2
votes
0answers
26 views

How does sampling works in case of imbalanced image datasets?

I am solving a problem of image classification of the image dataset for 3 classes. Dataset is highly imbalanced. How will sampling (either over- or under-sampling) work in that case? Should I remove (...
2
votes
0answers
25 views

How to feed key-value features (aggregated data) to LSTM?

I have the following time-series aggregated input for an LSTM-based model: ...
3
votes
1answer
94 views

How to fill missing values in a dataset where some properties can be inputs and outputs?

I have a dataset with missing values, I would like to use machine learning methods to fill. In more detail, there are $n$ individuals, for which up to 10 properties are provided, all numerical. The ...
1
vote
1answer
44 views

Are my steps correct for a proper classification of a sick brain?

I have a dataset with MRI of patients with a specific disease that affects the brain and another dataset with MRI of healthy patients. I want to create a classifier (using neural networks) to ...
1
vote
0answers
32 views

What pre-processing of the image is needed before feeding it into the convolutional neural network?

I can't figure out what preprocessing of the image is needed before feeding it into the convolutional neural network. For example, I want to recognize circles on a 1000 by 1000 px photo. The learning ...
0
votes
0answers
27 views

Do we scale our target feature in regression problems?

I know this is a basic problem, but still could not find answer, I feel like most books/tutorials avoid talking about scaling the output feature instead they just mention scaling input features. So ...
1
vote
0answers
32 views

Are there any general guidelines for dealing with imbalanced data through upsampling or downsampling?

Are there any general guidelines for dealing with imbalanced data through upsampling/downsampling? This Google developer guide suggests performing downsampling with upweighting, but for the most ...
1
vote
0answers
27 views

How can raw data from a motion sensor (like an IMU) reduced to the main points of the data

How can I reduce the caputured movement data of a person in a way, that I have filtered the main features of the movement. Or how can I detect pattern/ main features in that data set? I captured some ...
0
votes
0answers
55 views

How can I group the entries of the network traffic by their similarity?

I have the traffic of my network (with hundreds of entries). Below I am showing only 9 entries. ...
2
votes
0answers
16 views

How should I deal with multi-dimensional tensors for nodes in a graph convolution network?

How to work with GCN when the features of each node is not a 1D vector? For example, if the graph has N nodes and each node has features of the form $C \times D \times E$. Also, is there an open-...
2
votes
1answer
54 views

Do I need to denormalise results in linear regression?

I have learned so far how to linear regression with one or multiple features. So far, so good, everything seems to work fine, at least for my first simple examples. However, I now need to normalise ...
2
votes
0answers
38 views

Do I have to downsample the input and upsample the output of the neural network when implementing the NICE algorithm?

Consider that my input is an RGB image. The size of my image is $N\times N$. I'm trying to implement NICE algorithm presented by Dinh. The bijective function $f: \mathbb{R}^d \to \mathbb{R}^d$ maps $X$...
2
votes
0answers
22 views

What is the difference between training a model with RGB images and using only the color channels separately?

What is the difference between training a model with RGB images and using only the color channels separately (like only the red channel, green channel, etc.)? Would the model also learn patterns ...
1
vote
2answers
52 views

Is text preprocessing really all that necessary for NLP?

As a first step in many NLP courses, we learn about text preprocessing. The steps include lemmatization, removal of rare words, correcting typos etc. But I am not so sure about the actual ...
4
votes
1answer
80 views

How should I deal with variable-length inputs for neural networks?

I am a very beginner in the field of AI. I am basically a Pharma Professional without much coding experience. I use GUI based tools for the neural network. I am trying to develop an ANN that receives ...
2
votes
0answers
28 views

Can the addition of low-quality images to the training dataset increase the network performance?

I already trained a deep neural network called YOLO (You Only Look Once) with high-quality images (1920 by 1080 pixels) for a detection task. The result for mAP and IOU were 93% and 89% respectively. ...
4
votes
1answer
135 views

How should I deal with variable input sizes for a neural network classifier?

I am currently working on a project, where I have a sensor in a shoe that records the $X, Y, Z$ axes, from an acceleration and gyroscope sensor. Every millisecond, I get 6 data points. Now, the goal ...
1
vote
1answer
110 views

Is it recommended to remove stop words before named entity recognition?

Removing stop words can significantly speed up named entity recognition (NER) modeling by reducing the number of tokens in a document. Are stop words critical to get correct NER performance?
1
vote
1answer
59 views

Using three image datasets with different image sizes to train a CNN

I've just started with AI and CNN networks. I have two NIFTI images dataset, one with (240, 240) dimensions and the other one with (256, 132). Each dataset is front a different hospital and machine. ...
2
votes
0answers
17 views

Language Learning feedback with AI

Is there a program under development that uses AI technology, like Siri, to "hold hands" so to speak with a language learner and coach them on accent, colloqiual expressions, or to let them guide the ...
1
vote
1answer
34 views

Choosing Data Augmentation smartly for different application

I'm trying to understand the role of data augmentation and how it can affect the performance/accuracy of a deep model. My target application is fire detection (on video frames), with almost 15K ...
0
votes
1answer
39 views

How to automatically detect and correct false information in columnar data?

I'm working on data cleaning and I'm stuck. I have a data set with 3 columns: id, age, and weight. Supposing I have an entry: ...
1
vote
0answers
22 views

Acoustic Input Data: Decibel or Pascals

In acoustics decibel levels were defined to solve an issue with showing values that are interpretive, understandable, and easy to communicate in contrast to intensity or pressure in Pascals. $dB = ...
1
vote
0answers
15 views

EEG and Accelerometer Neural Network

I have frequency EEG data from fall and non-fall events and I am trying to incorporate it with accelerometer data that was collected at the same time. One approach is, of course, to use two separate ...
1
vote
0answers
13 views

Hashing images detects false duplicates

Like I said in the title, I used a code I found to identify duplicate images however there are many false alarms. Can someone give me a link to some code I could use that really works? The problem ...
2
votes
0answers
94 views

Suggestion for finding the stable regions in spiral galaxy data?

I am working with a data set that consists of the actual pitch angle (given as PA(Y)) and the pitch angle at each radii (listed from 1 to 217). In the image below, ...
2
votes
1answer
95 views

How to define the “Pre-Processing” in machine learning?

Is every process1 that is done on the data before we train the model is always called the pre-processing part? Or are there some processes which are not included? 1"Every process" is includes data ...
1
vote
0answers
30 views

Are there free and easy-to-use annotation tools for 3D bounding boxes?

Are there free and easy-to-use annotation tools for 3D bounding boxes? There is a lot of annotation tools out there, but most of them only good for 2d boxes.
1
vote
0answers
17 views

learning object recognition of primitive shapes through transfer learning problem

Question on transfer learning object classification (MobileNet_v2 with 75% number of parameters) with my own synthetic data: I made my own dataset of three shapes: triangles, rectangles and spheres. ...
1
vote
0answers
63 views

Focal loss for imbalanced multi class classification in Pytorch

I want an example code for Focal loss in PyTorch for a model with three class prediction. My model outputs 3 probabilities. ...
1
vote
1answer
45 views

How can I merge two datasets? [closed]

I want to merge 2 data sets in one, but don't know the right approach to do it. The datasets are similar, the last column is the same - will or not them buy a product. In the first dataset, users who ...
5
votes
1answer
284 views

What is “conditioning” on a feature?

On page 98 of Jet Substructure at the Large Hadron Collider: A Review of Recent Advances in Theory and Machine Learning the author writes; Redacted phase space: Studying the distribution of inputs ...
0
votes
0answers
16 views

What's the correct approach to standardise data from a time-series used for LSTM neural network predictions?

This question discusses the same model mentioned in Why is the value range of my LSTM model's prediction different from my test labels? Repeating the main points: I am using LSTM to do time-...
0
votes
4answers
100 views

How can I remove the noise from an EEG signal?

I am working on a project that takes signals from the brain and preprocesses them and then makes the machine learn about what human is thinking about. But I am struck in preprocessing of signal (...