Questions tagged [data-preprocessing]

For questions related to the concept of data pre-processing, which includes, for example, cleaning, instance selection, normalization, transformation, feature extraction or selection.

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

bad prediction when having noise on the data: LSTM time-series regression

I want to predict the force plate using a smart insole using the LSTM model for time series prediction. the data on the force plate has positive and negative values (I think the resulting positive ...
2 votes
0 answers
18 views

How to decide which column has more weightage to output

As per Image we can see Column_A value is directly proportional to output, While Change in value of Column_B has no effects in output. So basically I want to know is there any algorithm where I can ...
  • 21
0 votes
0 answers
22 views

How to handle missing data for an ordinal feature variable for a deep learning model?

BACKGROUND: I am developing a deep learning model in which one feature variable (out of many) is the grade of cancer, an ordinal variable. Below is a breakdown of my data by grade: ...
0 votes
0 answers
13 views

When training an LSTM, should you pad your dataset so the sequence length is static, or should it be variable?

I am putting together an LSTM network using visual basic. It's more of a learning exercise really, but it's also the only programming language I have access too at work. I am unsure of how to prepare ...
  • 323
0 votes
1 answer
37 views

How to encode categorical data for a convolutional model?

Is there a way to encode categorical nominal (no ordered) data to be used in CNN models? Let's say I need to create a 1D CNN model for categorization of time series but the values are not measurements,...
  • 1,220
0 votes
0 answers
26 views

How to filter point data based on pattern

I have a set of points represented as (x, y), and wish to separate out chunks/clusters of them which represent a ball trajectory. For example, in the two sample images below I wish to find out cluster ...
1 vote
1 answer
67 views

Feature Engineering on transactional dataset clustering

I have a data set with transactions details from different business (roughly 1 thousand business entities). Each row is a transaction. The structure of the dataset is as follows: client_id Sex Age ...
0 votes
0 answers
27 views

Retrieving data from a table by matching ID values

I have two columns [Uid, Con_id]; I need to fetch Uid id's which are matched with some Con_id's [From 606 to 615]. For example, all user_id's in each con_id from 606 to 615. But each Uid should match ...
0 votes
1 answer
103 views

combine two features in dataset?

I have a data set containing the number of security gaps and the level of that gap for a specific website. Now suppose I have 2 features in this data set, the first feature is the number of a ...
0 votes
1 answer
20 views

Adding several variables that could be important but can introduce overfitting

Sopose a productivity dataset, where day of the week and months day number are important. I'm thinking to encode these with a one-hot encoding. But if you have few years of data, that features might ...
0 votes
1 answer
18 views

How to encode both sentences and categorical data?

I have a DataFrame that contains several columns where some columns contain single words that can be category encoded since I know how many of them are there in total. However one column is an actual ...
  • 103
0 votes
0 answers
31 views

Raw Audio Data Learning with CNN: Use zero-centered Input with ReLu?

I am playing around with Conv-Nets on raw audio data. Found some papers that outline different architectures but did not find a lot about the data preprocessing. Can I use a zero-centred input for a ...
  • 1
0 votes
0 answers
12 views

Where have I gone wrong? Data Preprocessing and cross validation

I have the following steps: Fill in missing values - 'mean' for continuous, '10' for discrete columns (discrete columns go up to 0,1,2,3,4,5,6) Feature selection (correlation based, whole dataset) ...
0 votes
1 answer
40 views

How to detect and deal with data distribution drift/change?

I'm working on a problem in ML to assess the performance of multiple vendors. I have a set of features in my dataset, and it appears each vendor is characterized by its own distribution. This is my ...
0 votes
1 answer
81 views

Can I use discrete data in the same model as continuous data?

In my dataset, I have some data that is continuous - eg. Age and BMI. I also have some data that is discrete- for example, occupation is labelled as 1 ="Homemaker" 2="Working" 3=&...
0 votes
0 answers
41 views

Python code for Background label removal from Biomedical images

I am doing research in Biomedical Image processing and Deep Learning using Python language. I have labels in the background (as highlighted in yellow ) and Pectoral muscle ( as highlighted in red) of ...
0 votes
2 answers
30 views

Can I shuffle data for delivery duration forecast problem?

I'm new to ML and trying to write a solution to a food delivery duration time problem (so called lead time). I used algorithms such as random forest and gradient boosting which gave OK results but not ...
1 vote
0 answers
11 views

how to manage the impact of Covid on building a machine learning model

I need your suggestions for using historical data to build a machine learning model for analyzing the market and build an AI model(tree based model/random forest or regression analysis) for setting ...
  • 111
1 vote
2 answers
71 views

What makes a 'good' dataset

for the usage of ML technologies, having a appropriate dataset is arguably the first and fundamental step one has to tackle by either aquiring a dataset from external sources or creating their own. ...
  • 11
0 votes
0 answers
39 views

How to Extract Specific Information From Web Pages Containing Job Postings?

I am currently working on my first research project, which is to implement machine learning based web scraping in the e-recruitement environment. Currently, I am building the dataset that needs to be ...
0 votes
0 answers
13 views

How to train DINO on images with varying length (excluding padding options)?

I would like to train DINO (Emerging Properties in Self-Supervised Vision Transformers) on spectrograms of different size (specifically: different number of time bins and same number of frequency bins)...
  • 1
0 votes
1 answer
55 views

Is there a way to improve the low-quality data?

I'm on a robotics team and we've been tasked to write a program to differentiate between a live and dead fish. We've been given ~15 minutes of training footage and it's absolutely terrible. It's low ...
  • 15
2 votes
3 answers
381 views

Do I need to normalize all state-space variables? If so, how?

I am playing around with a DRL agent in a stock-trading environment. I have normalized all the external input data (the features that my agent will use). However, what about characteristics that don't ...
0 votes
1 answer
250 views

How should you reshape data before feeding it to LSTM layers?

I was curious if anyone had any advice on how to reshape data for a recurrent neural network. What I've been doing is array.reshape(len(X_train), # of points in time, # of features) And then in the ...
-1 votes
1 answer
35 views

Is this the right approach to preprocessing data for artificial neural-networks? [closed]

I recently participated in a competitive "hackathon" with the problem being binary classification of overall satisfaction for travelers. The dataset mostly consisted of survey questions and ...
0 votes
0 answers
24 views

Using an RNN for predicting columns of characters

I'm making an RNN using pytorch to learn from columns of tiles (each tile represented by a text character) and predict the next column of tiles. The training sequences are from maps of level data ...
0 votes
2 answers
60 views

How to represent multi-label colours in one-hot encoding?

Say I want to predict the price of a gemstone based on its colour. I have two options: averaging over its colour on an RGB scale, or using its textual description. If I was to choose the latter, how ...
5 votes
1 answer
112 views

Does the term "data augmentation" imply increasing the training dataset?

I have a manuscript that has been reviewed and one of the reviewers commented on my use of the term " data augmentation", saying that it might not be the appropriate term in my case (...
1 vote
3 answers
51 views

Why my classification results are correlated with the proportionality of my data?

I'm facing a problem. I'm working on mixed data model with NN (MLP & Word Embedding). My results are not pretty good. And I observed that the proportionality of my data are corelated with my ...
0 votes
1 answer
88 views

Normalizing float prices with movements up to a factor of 100

I have a bunch of arbitrary float numbers (asset prices), that I have to feed into a neural network. In the data set: values are between 1E-10 and 1E6 In a single sample: values may differ by a ...
0 votes
1 answer
39 views

How can my RNN get way better results than my ANN [closed]

So, I'm using the same dataset in both models but my RNN gets a 95% accuracy and my ANN gets 52%. It is a time series, binary classification problem, and I know that RNN is better than ANN for time ...
1 vote
0 answers
197 views

How do I deal with a dataset of Images with variable sizes (width and height) when doing Image Classification?

I have a dataset in which the images which don't have the same width and height. How do I perform Image Classification with such images? I am trying as much as possible to steer away from image ...
0 votes
0 answers
29 views

How can the agent be defined in a reinforcement learning problem with a tabular dataset as the environment?

Let's assume we need to train an RL model that drops duplicates in a tabular dataset? The actions should probably defined as drop or do nothing. But what should be the agent itself then? To me, it ...
  • 1
0 votes
1 answer
25 views

how to detect ouliers in audio dataset?

I'm currently working on an audio classification project using CNNs. The problem is I'm having trouble training my CNN. I doubt if there are outliers in my dataset but I don't know how to detect ...
0 votes
1 answer
51 views

Which generalization of standard deviation to use for multidimensional input normalization

For machine learning tasks, it's common to normalize input data by subtracting the mean $\mu$ and dividing by the standard deviation $\sigma$ of the dataset: $$\hat{x_i} = \frac{x_i - \mu}{\sigma}$$ —...
0 votes
0 answers
15 views

How to use tabular data and figures when the structure of data varies?

Suppose I have several journal articles. I would like to train a binary classifier on whether the journal article is insightful. NLP models such as BERT certainly fit my need by scanning the whole ...
1 vote
0 answers
28 views

Transforming Moving Position Data to an Inputvector for Neural Networks

Imagine a car is driving on a long street (= x-axis). The car can go in both directions and it will arbitrarily change its direction. I'm trying to formulate an Inputvector to tell a neural Network ...
  • 11
1 vote
1 answer
169 views

Which pre-processing steps are necessary for Deep Learning models to solve a document classification problem?

I have created a data set with 30.000 text documents (each text file is rather small with respect to its length), which are labelled with 0 and 1. Using this data set, I want to train machine learning ...
0 votes
0 answers
17 views

preprocessing of time series data, each line consist of a time series

Imagine having a dataset (almost 100000 observations) composed of 365 columns (1 year) and each index (or observation) will then be representing time-series data. In my case, each observation (time-...
0 votes
0 answers
7 views

Predicting single floats based on set of 2 feature arrays each of 100 values

I am trying to predict audio to video desynchronization based on ser od two arrays of lenght 100 which consist of coresponding audio and video samples. The problem is that my labels are single floats (...
1 vote
1 answer
50 views

Are derived or computed inputs bad for CNNs?

I am building a CNN and am wondering if inputting derived or computed inputs are generally bad for the effectiveness of CNNs? Or just NNs in general? By derived or computed values I mean data that is ...
1 vote
1 answer
145 views

General approaches in text encoding and labelling for NLP [closed]

What are the approaches of encoding text data? I would be glad to hear some summarization from experienced persons. And are there any solutions accepting words outside the vocabulary and including ...
  • 111
0 votes
1 answer
80 views

Can I flip a video to generate more data for action recognition?

There are 8 distinct action classes and around 50+ videos per class. I was wondering if flipping videos from the training set can be a good option to generate additional data. Is it?
0 votes
1 answer
96 views

How to arrange test dataset distribution for an imbalanced classification problem?

I have a dataset that contains 560 datapoints, and I would like to do binary classification on it. 400 datapoints belong to class 1, and 160 points belong to class 2. In the case of an imbalanced ...
  • 11
2 votes
1 answer
152 views

Best practice for handling letterboxed images for non fully-convolutional deep learning networks?

I'm working on a depth estimation network. It has two outputs: A relative depth map A scalar for scaling the relative depth map into an absolute depth map. This second output uses dense layers so ...
  • 153
2 votes
0 answers
35 views

What is the sensible amount of augmentation?

I am playing with the transforms from Torchvision. There are plenty of different kinds of these like: Resize RandomCrop ...
1 vote
1 answer
73 views

Does the converted (now square) distorted image of a face affect the accuracy of the calculation of the similarity in FaceNet?

As far as I know, FaceNet requires a square image as an input. MTCNN can detect and crop the original image as a square, but distortion occurs. Is it okay to feed the converted (now square) ...
  • 111
0 votes
1 answer
51 views

Given the immaturity of NLP tools for non-English languages, should I first translate the non-English language to English before text pre-processing?

For non-English languages (in my case Portuguese), what is the best approach? Should I use the not-so-complete tools in my language, or should I translate the text to English, and after using the ...
1 vote
0 answers
30 views

Recommended way to spilt image sequence for training/validation/testing

For object detection tasks I have a few minutes of video footage from a surveillance camera, converted to a sequence of images and ground truth bounding boxes for all people walking by. Now what's the ...
  • 33
1 vote
0 answers
48 views

Is having near-duplicates in a training dataset a bad thing?

I am making a labeled dataset of images from web streams for a CNN classification. Pictures from the same stream are quite similar as far as background, but slightly different as far as the main ...