Questions tagged [data-preprocessing]

For questions related to the concept of data pre-processing, which includes, for example, cleaning, instance selection, normalization, transformation, feature extraction or selection.

Filter by
Sorted by
Tagged with
0 votes
0 answers
5 views

How to train DINO on images with varying length (excluding padding options)?

I would like to train DINO (Emerging Properties in Self-Supervised Vision Transformers) on spectrograms of different size (specifically: different number of time bins and same number of frequency bins)...
user avatar
  • 1
0 votes
1 answer
40 views

Is there a way to improve the low-quality data?

I'm on a robotics team and we've been tasked to write a program to differentiate between a live and dead fish. We've been given ~15 minutes of training footage and it's absolutely terrible. It's low ...
user avatar
  • 1
2 votes
3 answers
107 views

Do I need to normalize all state-space variables? If so, how?

I am playing around with a DRL agent in a stock-trading environment. I have normalized all the external input data (the features that my agent will use). However, what about characteristics that don't ...
user avatar
0 votes
1 answer
37 views

How should you reshape data before feeding it to LSTM layers?

I was curious if anyone had any advice on how to reshape data for a recurrent neural network. What I've been doing is array.reshape(len(X_train), # of points in time, # of features) And then in the ...
user avatar
-1 votes
1 answer
25 views

Is this the right approach to preprocessing data for artificial neural-networks? [closed]

I recently participated in a competitive "hackathon" with the problem being binary classification of overall satisfaction for travelers. The dataset mostly consisted of survey questions and ...
user avatar
0 votes
0 answers
15 views

Using an RNN for predicting columns of characters

I'm making an RNN using pytorch to learn from columns of tiles (each tile represented by a text character) and predict the next column of tiles. The training sequences are from maps of level data ...
user avatar
0 votes
2 answers
42 views

How to represent multi-label colours in one-hot encoding?

Say I want to predict the price of a gemstone based on its colour. I have two options: averaging over its colour on an RGB scale, or using its textual description. If I was to choose the latter, how ...
user avatar
5 votes
1 answer
68 views

Does the term "data augmentation" imply increasing the training dataset?

I have a manuscript that has been reviewed and one of the reviewers commented on my use of the term " data augmentation", saying that it might not be the appropriate term in my case (...
user avatar
0 votes
1 answer
23 views

Why my classification results are correlated with the proportionality of my data?

I'm facing a problem. I'm working on mixed data model with NN (MLP & Word Embedding). My results are not pretty good. And I observed that the proportionality of my data are corelated with my ...
user avatar
0 votes
1 answer
81 views

Normalizing float prices with movements up to a factor of 100

I have a bunch of arbitrary float numbers (asset prices), that I have to feed into a neural network. In the data set: values are between 1E-10 and 1E6 In a single sample: values may differ by a ...
user avatar
0 votes
1 answer
36 views

How can my RNN get way better results than my ANN [closed]

So, I'm using the same dataset in both models but my RNN gets a 95% accuracy and my ANN gets 52%. It is a time series, binary classification problem, and I know that RNN is better than ANN for time ...
user avatar
1 vote
0 answers
28 views

How do I deal with a dataset of Images with variable sizes (width and height) when doing Image Classification?

I have a dataset in which the images which don't have the same width and height. How do I perform Image Classification with such images? I am trying as much as possible to steer away from image ...
user avatar
0 votes
0 answers
19 views

How can the agent be defined in a reinforcement learning problem with a tabular dataset as the environment?

Let's assume we need to train an RL model that drops duplicates in a tabular dataset? The actions should probably defined as drop or do nothing. But what should be the agent itself then? To me, it ...
user avatar
  • 1
0 votes
1 answer
19 views

how to detect ouliers in audio dataset?

I'm currently working on an audio classification project using CNNs. The problem is I'm having trouble training my CNN. I doubt if there are outliers in my dataset but I don't know how to detect ...
user avatar
0 votes
1 answer
42 views

Which generalization of standard deviation to use for multidimensional input normalization

For machine learning tasks, it's common to normalize input data by subtracting the mean $\mu$ and dividing by the standard deviation $\sigma$ of the dataset: $$\hat{x_i} = \frac{x_i - \mu}{\sigma}$$ —...
user avatar
0 votes
0 answers
12 views

How to use tabular data and figures when the structure of data varies?

Suppose I have several journal articles. I would like to train a binary classifier on whether the journal article is insightful. NLP models such as BERT certainly fit my need by scanning the whole ...
user avatar
1 vote
0 answers
28 views

Transforming Moving Position Data to an Inputvector for Neural Networks

Imagine a car is driving on a long street (= x-axis). The car can go in both directions and it will arbitrarily change its direction. I'm trying to formulate an Inputvector to tell a neural Network ...
user avatar
  • 11
1 vote
1 answer
103 views

Which pre-processing steps are necessary for Deep Learning models to solve a document classification problem?

I have created a data set with 30.000 text documents (each text file is rather small with respect to its length), which are labelled with 0 and 1. Using this data set, I want to train machine learning ...
user avatar
0 votes
0 answers
17 views

preprocessing of time series data, each line consist of a time series

Imagine having a dataset (almost 100000 observations) composed of 365 columns (1 year) and each index (or observation) will then be representing time-series data. In my case, each observation (time-...
user avatar
0 votes
0 answers
7 views

Predicting single floats based on set of 2 feature arrays each of 100 values

I am trying to predict audio to video desynchronization based on ser od two arrays of lenght 100 which consist of coresponding audio and video samples. The problem is that my labels are single floats (...
user avatar
1 vote
1 answer
44 views

Are derived or computed inputs bad for CNNs?

I am building a CNN and am wondering if inputting derived or computed inputs are generally bad for the effectiveness of CNNs? Or just NNs in general? By derived or computed values I mean data that is ...
user avatar
1 vote
1 answer
45 views

General approaches in text encoding and labelling for NLP [closed]

What are the approaches of encoding text data? I would be glad to hear some summarization from experienced persons. And are there any solutions accepting words outside the vocabulary and including ...
user avatar
  • 111
0 votes
1 answer
59 views

Can I flip a video to generate more data for action recognition?

There are 8 distinct action classes and around 50+ videos per class. I was wondering if flipping videos from the training set can be a good option to generate additional data. Is it?
user avatar
0 votes
1 answer
32 views

How to arrange test dataset distribution for an imbalanced classification problem?

I have a dataset that contains 560 datapoints, and I would like to do binary classification on it. 400 datapoints belong to class 1, and 160 points belong to class 2. In the case of an imbalanced ...
user avatar
  • 11
2 votes
1 answer
67 views

Best practice for handling letterboxed images for non fully-convolutional deep learning networks?

I'm working on a depth estimation network. It has two outputs: A relative depth map A scalar for scaling the relative depth map into an absolute depth map. This second output uses dense layers so ...
user avatar
  • 153
2 votes
0 answers
27 views

What is the sensible amount of augmentation?

I am playing with the transforms from Torchvision. There are plenty of different kinds of these like: Resize RandomCrop ...
user avatar
1 vote
1 answer
38 views

Does the converted (now square) distorted image of a face affect the accuracy of the calculation of the similarity in FaceNet?

As far as I know, FaceNet requires a square image as an input. MTCNN can detect and crop the original image as a square, but distortion occurs. Is it okay to feed the converted (now square) ...
user avatar
  • 111
0 votes
1 answer
36 views

Given the immaturity of NLP tools for non-English languages, should I first translate the non-English language to English before text pre-processing?

For non-English languages (in my case Portuguese), what is the best approach? Should I use the not-so-complete tools in my language, or should I translate the text to English, and after using the ...
user avatar
0 votes
0 answers
31 views

Correct way for validation split in semi-supervised learning

I have a fully labeled Human Activity Recognition dataset from 9 participants. I am using one participant for the test set, and for the training, I use one participant as a labeled set and the rest as ...
user avatar
  • 31
0 votes
0 answers
17 views

What are the pros and cons of using a normal positional encoding in an adjacency matrix?

I understand that a normal positional encoding helps a transformer to understand pictures better and that it allows the (otherwise permutational invariant transformer-network) to create relationships ...
user avatar
0 votes
0 answers
16 views

What is an efficient way to implement an contextual search for a search engine?

So I am working on a search engine and want to implement a contextual search for my search engine but would like to get some advice before I start working on it. I have a basic knowledge of A.I. like ...
user avatar
0 votes
0 answers
11 views

How reasonable is it to predict on a variable log scale?

Hi I know this is a super common practice and it seems reasonable to me. But I've actually encountered a problem recently where the benchmark MAE to beat was in exponential domain and naturally a fair ...
user avatar
  • 276
1 vote
0 answers
27 views

Recommended way to spilt image sequence for training/validation/testing

For object detection tasks I have a few minutes of video footage from a surveillance camera, converted to a sequence of images and ground truth bounding boxes for all people walking by. Now what's the ...
user avatar
  • 33
1 vote
0 answers
41 views

Is having near-duplicates in a training dataset a bad thing?

I am making a labeled dataset of images from web streams for a CNN classification. Pictures from the same stream are quite similar as far as background, but slightly different as far as the main ...
user avatar
1 vote
0 answers
39 views

Is it true that real world data is highly discontinuous?

A function $f$ is said to be continuous at a point $c$ if it satisfies three properties: Should be defined at the point $c$ Left and right-hand limits at $c$ must be equal i.e., the limit must exist ...
user avatar
  • 3,099
0 votes
1 answer
26 views

Data analysis before feeding to ML pipeline

I'm new to machine learning and I've been working through a dataset of ~3000 records with ~100 features. I've been hand rolling Python and R scripts to analyse the data. For example, plotting the ...
user avatar
  • 103
1 vote
1 answer
36 views

How can I address missing values for LSTM?

I'm a student and writing my first paper for submission on conference. I have a question there is a dataset below. this is temporal-spatial dataset. ...
user avatar
0 votes
0 answers
18 views

How should an ML model architecture be designed for predicting the order of a sequence?

I've decided to create a model for predicting Formula 1 race results based on driver statistics, to try to improve my ML skills. The first problem I've encountered is the data type of the target ...
user avatar
  • 1
1 vote
1 answer
23 views

What to do when you have massive amount of data but you don't have enough computation power for training a machine learning model?

For example, I have a massive amount of data, but I have limited computational resources and time to train on the full data. Other cases may include, I have huge amounts of 360-degree images, where I ...
user avatar
  • 111
1 vote
1 answer
161 views

Rescaling time-series data with very spiky pattern for training data in LSTM network

I am working with some time-series hydrology data. Our goal is to forecast the time series forward, meaning predicting the data 1 month, 3 months ,6 months into the future. The data itself(image below)...
user avatar
  • 187
0 votes
0 answers
32 views

Do the training and test datasets need to be equally preprocessed as one whole dataset?

I have developed, trained and tested an NLP model. It is persisted in a pickle file. The model contains the data preprocessing function that includes text cleaning and new features engineered with ...
user avatar
  • 101
0 votes
1 answer
31 views

Normalization of possibly not fully representative data

I am trying to train a classification RNN model on a sequence of table medical data, but I stuck with the normalization problem. I realized that I cannot simply use MinMaxScaler, because of 3 problems:...
user avatar
1 vote
2 answers
90 views

Is creating dataset only by augmentation a bad practice?

I wonder if creating data set only by augmentation base images is a bad practice. I mean the situation when you have to train net to predict really simple patterns, for example printed-like digits. ...
user avatar
2 votes
5 answers
156 views

How do I increase the size of an (almost) balanced dataset?

I am trying to add more data points in my (almost) balanced dataset for training my neural network. I have come across techniques such as SMOTE or Random Over Sampling, but they work best for ...
user avatar
2 votes
1 answer
54 views

How to handle class imbalance when the actual data are that way

My supervised learning training data are obtained from actual data; and in real cases, there's one class that happens less often than other classes, just around 5% of all cases. To be precise, the ...
user avatar
  • 1,183
1 vote
0 answers
31 views

Why do you calculate the mean and standard deviation over the complete dataset before training rather than for every batch?

In most implementations of neural networks the features are scaled to make the optimization of the loss function as stable as possible. Mostly a min-max scaler is used. Alternatively, there is also a ...
user avatar
  • 11
0 votes
0 answers
66 views

Why (not) using pre-processing before using Transformer models?

Regarding the use of pre-processing techniques before using Transformers models, I read this post that apparently says that these measures are not so necessary nor interfere so much in the final ...
user avatar
0 votes
1 answer
64 views

How should I incorporate numerical and categorical data as part of the inputs to the U-net for semantic segmentation?

I am using a U-Net to segment cancer cells in images of patients' arms. I would like to add patient data to it in order to see if it is possible to enhance the segmentation (patient data comes in the ...
user avatar
  • 115
0 votes
2 answers
102 views

Is my approach to building an RNN to predict the probability that the word is in English appropriate?

Goal To build an RNN which would receive a word as an input, and output the probability that the word is in English (or at least would be English sounding). Example ...
user avatar
2 votes
2 answers
96 views

Is feature engineer an important step for a deep learning approach?

I'd like to ask you if feature engineering is an important step for a deep learning approach. By feature engineering I mean some advanced preprocessing steps, such as looking at histogram ...
user avatar
  • 509