Questions tagged [data-preprocessing]

For questions related to the concept of data pre-processing, which includes, for example, cleaning, instance selection, normalization, transformation, feature extraction or selection.

Filter by
Sorted by
Tagged with
1
vote
1answer
40 views

How to normalize images before training?

I have seen people normalize images by just dividing 255. But why? Why not use mean normalization or Z-score Normalization? I also came across this StackOverflow topic while searching but the answers ...
0
votes
0answers
17 views

Includes only some features of the dataset in a dataloader [closed]

I created a dataset and to creat a dataloader with Pytorch. I have the following problem, I would like to create a dataloader with only the features data and y_vector. As I apply torch.utils.data....
1
vote
1answer
24 views

Does feature scaling have any benefits if all features are on the same scale?

By scaling features, we can prevent one feature from dominating the decisions of a model. For example, say heights (cm), and age (years) are two features in my data. Since range of heights is larger ...
1
vote
1answer
35 views

Is data leakage relevant when scaling across samples?

I have a question about data leakage when pre-processing data for a neural network and whether data leakage actually applies in my instance. I have variance stabilising transformed genomic data. ...
0
votes
0answers
33 views

Data Augmentation of store images using handwritten labels

I am new to AI and NN. I've started learning using Geron's book on Tensorflow. My first project ("Smart Shelf") is to determine which items in a store have been purchased and need refilled. ...
0
votes
1answer
36 views

Denoising Images When Training a Classification Model

Suppose you have a binary outcome variable and have some training data (10,000 images in jpg format). Also you have a test set of say 11,000 images. If we want to train a classification model and want ...
0
votes
0answers
13 views

Low Signal-To-Noise Ratio Data Processing and Model Choices

Attached is a plot showing that y regresses on individual features: Scatter Plot And as shown, each feature has minuscule predictive power on y. I have 130 features like this, all of them are more or ...
3
votes
2answers
45 views

Is pre-processing used in deep learning?

I'm new to deep learning. I wanted to know: do we use pre-processing in deep learning? Or it is only used in machine learning. I searched for it and its methods on the internet, but I didn't find a ...
0
votes
1answer
64 views

How to predict the best from a set of messages - best practice

Suppose I have a set of messages A,B,C,D and I want to produce the best message for a website user at a given time. For training I plan to show random users a random single message [A/B/C/D] and fill ...
2
votes
1answer
72 views

In OCR, how should I deal with the warped text on the sides of oval objects?

Consider an image that contains one can (or bottle, or any similar oval object), which has texts all over it. In the image below, I have many bottles, but you can assume that each image only contains ...
2
votes
1answer
63 views

How robust are deep networks to class imbalance?

Before deep learning, I worked with machine learning problems where the data had a large class imbalance (30:1 or worse ratios). At that time, all the classifiers struggled, even after under-sampling ...
0
votes
0answers
21 views

Is it possible to automatically remove ignore (or remove) the equations (and other noisy elements) while performing OCR?

I have academic pdf data. I am using OCR for converting it into text format. The pdf has a few mathematical equations and terms which are acting as noise for my task. Any way through which the task of ...
0
votes
1answer
108 views

How to fill NaNs in Cross-Validation?

I have been searching this but did not find the answer, so sorry if this is a duplicated question. I was working with cross-validation, where some doubts came to my mind, and I am not sure about which ...
2
votes
0answers
54 views

Statistical method for selecting features for classification

I'm working on a classifier for the famous MNIST handwritten data set. I want to create a few features on my own, and I want to be able to estimate which feature might perform better before actually ...
0
votes
0answers
29 views

How to generate a matrix out of sparse data?

I have a system that takes 32 inputs (all of which are 1 or 0) and generates 32 outputs (all of which are complex numbers that lie roughly in the range of (0,2)). The response of this system to its ...
0
votes
1answer
23 views

Should binary feature be in one or two columns in deep neural networks?

Let's assume I have a simple feedforward neural network whose input contains binary 0/1 features and output is also binary two classes. Is it better, worse or maybe totally indifferent, for every of ...
1
vote
1answer
53 views

Why do we resize images before using them for object detection?

In object detection, we can resize images by keeping the ratio the same as the original image, which is often known as "letterbox" resize. My questions are Why do we need to resize images? ...
0
votes
0answers
31 views

Transforming neural network target values before training

Consider the scenario in which I am measuring certain $f(a,x)$, which i want to be the target value for some related input $g(a,x)$. In other words, I am trying to map $$g(a,x)\Rightarrow f(a,x)$$ I ...
1
vote
1answer
92 views

How to mathematically describe the convolution operation (with a Gaussian kernel)?

I have to build a model where I pre-process the data with a Gaussian kernel. The data are an $n\times n$ matrix (i.e one channel), but not an image, thus I can't refer to this matrix as an image and ...
0
votes
1answer
36 views

What would be a typical pre-processing and data normalization pipeline for time series data (for non-linear models such as neural networks)?

I've started to work on time series. I was wondering what would be the best data normalizing and pre-processing technique for non-linear models, specifically, neural networks. One I can think of is ...
1
vote
0answers
14 views

Multiple Inertia sensors system based for gestures recognition

I am a newbie to Machine Learning field as I am engaging to a personal project that I am trying to use the 6 degree of freedom Inertial Measurement Units(IMUs) measuring the Acceleration acting on 3 ...
2
votes
1answer
44 views

Should I remove the text overlaying some images in the dataset before training the CNN?

If I am attempting to train a CNN on some image data to perform image classification, but some of the images have pieces of text overlaying them (for the purpose of description to humans), then is it ...
0
votes
1answer
23 views

Can a neural network be trained on a dataset containing only values for true output for a classification problem?

I am using a dataset from Google which contains 1,27,000 data points on simulated concentrations of the atmosphere of exoplanets which can sustain life. So, the output label of all these data points ...
3
votes
1answer
72 views

How to perform prediction when some features have missing values?

Sorry if this is too noob question, I'm just a beginner. I have a data set with companies' info. There are 2 kinds of features: financial (revenue and so on) and general info (like the number of ...
1
vote
0answers
52 views

How much should we augment our training data?

I am wondering how much I should extend my training set with data augmentation. Is there somewhere a pre-defined number I can go with? Suppose I have 10000 images, can I go as far as 10x or 20x times, ...
0
votes
1answer
52 views

Do the rows of the design matrix refer to the observations or predictors?

I attempt to understand the formulation of dictionary learning for this paper: Depression Detection via Harvesting Social Media: A Multimodal Dictionary Learning Solution Multimodal Task-Driven ...
2
votes
3answers
73 views

When to convert data to word embeddings in NLP

When training a network using word embeddings, it is standard to add an embedding layer to first convert the input vector to the embeddings. However, assuming the embeddings are pre-trained and frozen,...
2
votes
2answers
37 views

How to take the optimal batch_size for training a model?

I have an image dataset, which is composed of 113695 images for training and 28424 images for validation. Now, when I use ImageDataGenerator and ...
1
vote
0answers
138 views

Can I resize my images after labeling them?

Is it okay if I label my images with their original size and then resize them, or should I first resize them and then label them? I mean do I need to recalibrate my labels if I resized my images?
0
votes
0answers
103 views

Multilabel stratified split for images/object detection

I am working on an object detection model and have thought of looking into stratified splits for the dataset. Now since I am doing object detection I have a variable number of "labels" for ...
0
votes
1answer
62 views

Is it necessary to standardise the expected output

Normalisation transform data into a range: $$X_i = \dfrac{X_i - Min}{Max-Min}$$ Practically, I found out that the model doesn't generalise well when using normalisation of input data, instead of ...
1
vote
1answer
59 views

Do I need to rotate the masks, if I also rotate the images and the masks are generated from the input?

I am training a neural network that takes an input (H, W, 3) and has the output of size (H', W', C). Now, to augment my dataset, ...
1
vote
0answers
22 views

How is the performance of a CNN trained with monochrome images on image recognition tasks degraded?

For CNN image recognition tasks, like object recognition/face recognition/object segmentation/posture recognition, are there experiment results about how much will the performance be degraded with ...
1
vote
0answers
33 views

How can I predict the label given a partial feature vector?

Most of the traditional machine learning algorithms need a feature vector of a constant dimension to predict the label. Which algorithms can be used to predict a class label with a shorter or ...
2
votes
2answers
71 views

How to train a neural network with a data set that in which the target is a mix of 0-1 label and numeric real value label?

I am running into an issue in which the the target (label collums) of my dataset contain a mixture of binary label (yes/no) and some numeric value label. The value of these numeric value (resource 1 ...
1
vote
1answer
53 views

Can I find a mapping that minimizes the maximum distance ratio of certain vectors?

Let's say we have several vector points. My goal is to distinguish the vectors, so I want to make them far from each other. Some of them are already far from each other, but some of them can be ...
2
votes
0answers
26 views

How does sampling works in case of imbalanced image datasets?

I am solving a problem of image classification of the image dataset for 3 classes. Dataset is highly imbalanced. How will sampling (either over- or under-sampling) work in that case? Should I remove (...
2
votes
0answers
27 views

How to feed key-value features (aggregated data) to LSTM?

I have the following time-series aggregated input for an LSTM-based model: ...
4
votes
1answer
98 views

How to fill missing values in a dataset where some properties can be inputs and outputs?

I have a dataset with missing values, I would like to use machine learning methods to fill. In more detail, there are $n$ individuals, for which up to 10 properties are provided, all numerical. The ...
1
vote
1answer
45 views

Are my steps correct for a proper classification of a sick brain?

I have a dataset with MRI of patients with a specific disease that affects the brain and another dataset with MRI of healthy patients. I want to create a classifier (using neural networks) to ...
1
vote
0answers
35 views

What pre-processing of the image is needed before feeding it into the convolutional neural network?

I can't figure out what preprocessing of the image is needed before feeding it into the convolutional neural network. For example, I want to recognize circles on a 1000 by 1000 px photo. The learning ...
0
votes
0answers
27 views

Do we scale our target feature in regression problems?

I know this is a basic problem, but still could not find answer, I feel like most books/tutorials avoid talking about scaling the output feature instead they just mention scaling input features. So ...
1
vote
0answers
43 views

Are there any general guidelines for dealing with imbalanced data through upsampling or downsampling?

Are there any general guidelines for dealing with imbalanced data through upsampling/downsampling? This Google developer guide suggests performing downsampling with upweighting, but for the most ...
1
vote
0answers
27 views

How can raw data from a motion sensor (like an IMU) reduced to the main points of the data

How can I reduce the caputured movement data of a person in a way, that I have filtered the main features of the movement. Or how can I detect pattern/ main features in that data set? I captured some ...
0
votes
0answers
55 views

How can I group the entries of the network traffic by their similarity?

I have the traffic of my network (with hundreds of entries). Below I am showing only 9 entries. ...
2
votes
0answers
16 views

How should I deal with multi-dimensional tensors for nodes in a graph convolution network?

How to work with GCN when the features of each node is not a 1D vector? For example, if the graph has N nodes and each node has features of the form $C \times D \times E$. Also, is there an open-...
2
votes
1answer
62 views

Do I need to denormalise results in linear regression?

I have learned so far how to linear regression with one or multiple features. So far, so good, everything seems to work fine, at least for my first simple examples. However, I now need to normalise ...
2
votes
0answers
46 views

Do I have to downsample the input and upsample the output of the neural network when implementing the NICE algorithm?

Consider that my input is an RGB image. The size of my image is $N\times N$. I'm trying to implement NICE algorithm presented by Dinh. The bijective function $f: \mathbb{R}^d \to \mathbb{R}^d$ maps $X$...
2
votes
0answers
25 views

What is the difference between training a model with RGB images and using only the color channels separately?

What is the difference between training a model with RGB images and using only the color channels separately (like only the red channel, green channel, etc.)? Would the model also learn patterns ...
1
vote
2answers
62 views

Is text preprocessing really all that necessary for NLP?

As a first step in many NLP courses, we learn about text preprocessing. The steps include lemmatization, removal of rare words, correcting typos etc. But I am not so sure about the actual ...