Questions tagged [data-preprocessing]

For questions related to the concept of data pre-processing, which includes, for example, cleaning, instance selection, normalization, transformation, feature extraction or selection.

Filter by
Sorted by
Tagged with
0
votes
0answers
18 views

Pandas dataframe to keras LSTM input shape [closed]

I have a simple dataset consisting of only two columns (year and price of oil). Now, I need to shape them in order for keras' LSTM-layer to accept their input_shape. My code looks like this, I ...
1
vote
1answer
25 views

How can I address missing values for LSTM?

I'm a student and writing my first paper for submission on conference. I have a question there is a dataset below. this is temporal-spatial dataset. ...
0
votes
0answers
16 views

How should an ML model architecture be designed for predicting the order of a sequence?

I've decided to create a model for predicting Formula 1 race results based on driver statistics, to try to improve my ML skills. The first problem I've encountered is the data type of the target ...
1
vote
1answer
19 views

What to do when you have massive amount of data but you don't have enough computation power for training a machine learning model?

For example, I have a massive amount of data, but I have limited computational resources and time to train on the full data. Other cases may include, I have huge amounts of 360-degree images, where I ...
0
votes
0answers
10 views

Should we create a label mean group of small nearby objects in object detection?

I'm working on object detection models and my dataset sometimes has a lot of small objects (stay far from the scene) (overlapping and nearby) which is really annoying in annotating (it's too small and ...
0
votes
0answers
20 views

What is a good approach to apply smoothing techniques to time series with big changes and seasonality?

As far as I understand, it is always a good idea to apply a smoothing technique to a raw time series before training a model with it. However, I have a time series with big changes in magnitude and ...
1
vote
1answer
38 views

Rescaling time-series data with very spiky pattern for training data in LSTM network

I am working with some time-series hydrology data. Our goal is to forecast the time series forward, meaning predicting the data 1 month, 3 months ,6 months into the future. The data itself(image below)...
0
votes
0answers
27 views

Do the training and test datasets need to be equally preprocessed as one whole dataset?

I have developed, trained and tested an NLP model. It is persisted in a pickle file. The model contains the data preprocessing function that includes text cleaning and new features engineered with ...
0
votes
1answer
25 views

Normalization of possibly not fully representative data

I am trying to train a classification RNN model on a sequence of table medical data, but I stuck with the normalization problem. I realized that I cannot simply use MinMaxScaler, because of 3 problems:...
0
votes
0answers
9 views

Preprocessing deterministic data with sklearn

I am trying to create a set of ML models that will serve as a replacement for a complex deterministic simulation. The simulation requires 4 inputs (x1, x2, x3 and x4) to determine 4 different outputs (...
1
vote
2answers
53 views

Is creating dataset only by augmentation a bad practice?

I wonder if creating data set only by augmentation base images is a bad practice. I mean the situation when you have to train net to predict really simple patterns, for example printed-like digits. ...
0
votes
0answers
90 views

How does pairwise comparison training work in XGBoost's XGBRanker?

I'm interested in learning to rank with pairwise comparison. While working on this, I found that XGBoost has a model called XGBRanker, which works very well. I want to find out how the XGBRanker ...
2
votes
5answers
70 views

How do I increase the size of an (almost) balanced dataset?

I am trying to add more data points in my (almost) balanced dataset for training my neural network. I have come across techniques such as SMOTE or Random Over Sampling, but they work best for ...
2
votes
1answer
32 views

How to handle class imbalancing when the actual data are that way

My supervised learning training data are obtained from actual data; and in real cases, there's one class which happens less often than other classes, just around 5% of all cases. To be precise, the ...
1
vote
0answers
27 views

Why do you calculate the mean and standard deviation over the complete dataset before training rather than for every batch?

In most implementations of neural networks the features are scaled to make the optimization of the loss function as stable as possible. Mostly a min-max scaler is used. Alternatively, there is also a ...
0
votes
0answers
16 views

Can I take some conclusion from two dimension PCA

I have a table of 140 features and try to predict binary classification. I create two dimensions out of this for visualization with PCA. In this visualization I can see, that data is not separable but ...
0
votes
0answers
15 views

Dealing with huge peak in data distribution

I am trying to predict a continuous value using a deep neural network. I have about 100,000 samples, where input is a sequence of RNA, and output is a continuous metric determining the quality of the ...
0
votes
0answers
19 views

Why (not) using pre-processing before using Transformer models?

Regarding the use of pre-processing techniques before using Transformers models, I read this post that apparently says that these measures are not so necessary nor interfere so much in the final ...
0
votes
1answer
29 views

How should I incorporate numerical and categorical data as part of the inputs to the U-net for semantic segmentation?

I am using a U-Net to segment cancer cells in images of patients' arms. I would like to add patient data to it in order to see if it is possible to enhance the segmentation (patient data comes in the ...
0
votes
2answers
86 views

Is my approach to building an RNN to predict the probability that the word is in English appropriate?

Goal To build an RNN which would receive a word as an input, and output the probability that the word is in English (or at least would be English sounding). Example ...
2
votes
2answers
61 views

Is feature engineer an important step for a deep learning approach?

I'd like to ask you if feature engineering is an important step for a deep learning approach. By feature engineering I mean some advanced preprocessing steps, such as looking at histogram ...
2
votes
0answers
39 views

How to deal with a variable number of channels of the inputs?

I have a problem in which my input data may have a varying number of channels. Let me explain with an example. Imagine we have a classification problem in which we wish to identify if certain species ...
1
vote
1answer
48 views

How to normalize images before training?

I have seen people normalize images by just dividing 255. But why? Why not use mean normalization or Z-score Normalization? I also came across this StackOverflow topic while searching but the answers ...
1
vote
1answer
26 views

Does feature scaling have any benefits if all features are on the same scale?

By scaling features, we can prevent one feature from dominating the decisions of a model. For example, say heights (cm), and age (years) are two features in my data. Since range of heights is larger ...
2
votes
1answer
42 views

Is data leakage relevant when scaling across samples?

I have a question about data leakage when pre-processing data for a neural network and whether data leakage actually applies in my instance. I have variance stabilising transformed genomic data. ...
0
votes
0answers
35 views

Data Augmentation of store images using handwritten labels

I am new to AI and NN. I've started learning using Geron's book on Tensorflow. My first project ("Smart Shelf") is to determine which items in a store have been purchased and need refilled. ...
0
votes
1answer
44 views

Denoising Images When Training a Classification Model

Suppose you have a binary outcome variable and have some training data (10,000 images in jpg format). Also you have a test set of say 11,000 images. If we want to train a classification model and want ...
0
votes
0answers
15 views

Low Signal-To-Noise Ratio Data Processing and Model Choices

Attached is a plot showing that y regresses on individual features: Scatter Plot And as shown, each feature has minuscule predictive power on y. I have 130 features like this, all of them are more or ...
3
votes
2answers
50 views

Is pre-processing used in deep learning?

I'm new to deep learning. I wanted to know: do we use pre-processing in deep learning? Or it is only used in machine learning. I searched for it and its methods on the internet, but I didn't find a ...
0
votes
1answer
65 views

How to predict the best from a set of messages - best practice

Suppose I have a set of messages A,B,C,D and I want to produce the best message for a website user at a given time. For training I plan to show random users a random single message [A/B/C/D] and fill ...
2
votes
1answer
103 views

In OCR, how should I deal with the warped text on the sides of oval objects?

Consider an image that contains one can (or bottle, or any similar oval object), which has texts all over it. In the image below, I have many bottles, but you can assume that each image only contains ...
2
votes
1answer
72 views

How robust are deep networks to class imbalance?

Before deep learning, I worked with machine learning problems where the data had a large class imbalance (30:1 or worse ratios). At that time, all the classifiers struggled, even after under-sampling ...
0
votes
0answers
22 views

Is it possible to automatically remove ignore (or remove) the equations (and other noisy elements) while performing OCR?

I have academic pdf data. I am using OCR for converting it into text format. The pdf has a few mathematical equations and terms which are acting as noise for my task. Any way through which the task of ...
0
votes
1answer
118 views

How to fill NaNs in Cross-Validation?

I have been searching this but did not find the answer, so sorry if this is a duplicated question. I was working with cross-validation, where some doubts came to my mind, and I am not sure about which ...
2
votes
0answers
60 views

Statistical method for selecting features for classification

I'm working on a classifier for the famous MNIST handwritten data set. I want to create a few features on my own, and I want to be able to estimate which feature might perform better before actually ...
0
votes
0answers
30 views

How to generate a matrix out of sparse data?

I have a system that takes 32 inputs (all of which are 1 or 0) and generates 32 outputs (all of which are complex numbers that lie roughly in the range of (0,2)). The response of this system to its ...
1
vote
1answer
35 views

Should binary feature be in one or two columns in deep neural networks?

Let's assume I have a simple feedforward neural network whose input contains binary 0/1 features and output is also binary two classes. Is it better, worse, or maybe totally indifferent, for every ...
1
vote
1answer
101 views

Why do we resize images before using them for object detection?

In object detection, we can resize images by keeping the ratio the same as the original image, which is often known as "letterbox" resize. My questions are Why do we need to resize images? ...
0
votes
0answers
34 views

Transforming neural network target values before training

Consider the scenario in which I am measuring certain $f(a,x)$, which i want to be the target value for some related input $g(a,x)$. In other words, I am trying to map $$g(a,x)\Rightarrow f(a,x)$$ I ...
1
vote
1answer
118 views

How to mathematically describe the convolution operation (with a Gaussian kernel)?

I have to build a model where I pre-process the data with a Gaussian kernel. The data are an $n\times n$ matrix (i.e one channel), but not an image, thus I can't refer to this matrix as an image and ...
0
votes
1answer
43 views

What would be a typical pre-processing and data normalization pipeline for time series data (for non-linear models such as neural networks)?

I've started to work on time series. I was wondering what would be the best data normalizing and pre-processing technique for non-linear models, specifically, neural networks. One I can think of is ...
1
vote
0answers
14 views

Multiple Inertia sensors system based for gestures recognition

I am a newbie to Machine Learning field as I am engaging to a personal project that I am trying to use the 6 degree of freedom Inertial Measurement Units(IMUs) measuring the Acceleration acting on 3 ...
2
votes
1answer
47 views

Should I remove the text overlaying some images in the dataset before training the CNN?

If I am attempting to train a CNN on some image data to perform image classification, but some of the images have pieces of text overlaying them (for the purpose of description to humans), then is it ...
0
votes
1answer
23 views

Can a neural network be trained on a dataset containing only values for true output for a classification problem?

I am using a dataset from Google which contains 1,27,000 data points on simulated concentrations of the atmosphere of exoplanets which can sustain life. So, the output label of all these data points ...
3
votes
1answer
76 views

How to perform prediction when some features have missing values?

Sorry if this is too noob question, I'm just a beginner. I have a data set with companies' info. There are 2 kinds of features: financial (revenue and so on) and general info (like the number of ...
1
vote
0answers
82 views

How much should we augment our training data?

I am wondering how much I should extend my training set with data augmentation. Is there somewhere a pre-defined number I can go with? Suppose I have 10000 images, can I go as far as 10x or 20x times, ...
0
votes
1answer
55 views

Do the rows of the design matrix refer to the observations or predictors?

I attempt to understand the formulation of dictionary learning for this paper: Depression Detection via Harvesting Social Media: A Multimodal Dictionary Learning Solution Multimodal Task-Driven ...
3
votes
3answers
79 views

When to convert data to word embeddings in NLP

When training a network using word embeddings, it is standard to add an embedding layer to first convert the input vector to the embeddings. However, assuming the embeddings are pre-trained and frozen,...
2
votes
2answers
41 views

How to take the optimal batch_size for training a model?

I have an image dataset, which is composed of 113695 images for training and 28424 images for validation. Now, when I use ImageDataGenerator and ...
1
vote
0answers
322 views

Can I resize my images after labeling them?

Is it okay if I label my images with their original size and then resize them, or should I first resize them and then label them? I mean do I need to recalibrate my labels if I resized my images?