Questions tagged [data-preprocessing]
For questions related to the concept of data pre-processing, which includes, for example, cleaning, instance selection, normalization, transformation, feature extraction or selection.
159
questions
0
votes
0
answers
6
views
Linking preprocessing with models
We have been working on a deep learning problem for a few iterations now. We've been tweaking the preprocessing as we go. We've also been training models as we go.
The people using the models for ...
- 125
1
vote
1
answer
13
views
Trying to Replicate the Pile - pass2_shuffle_holdout.py - ModuleNotFoundError: No module named 'parse' [closed]
Goal
I'm trying to replicate a subset of the Pile that works with their GPT-NeoX trainer. I have pretty good hardware, but nothing like the 90 Tesla rack that made the 20 billion parameter GPT-NeoX-20 ...
- 111
0
votes
0
answers
11
views
What computer vision techniques can help me select the best image for object detection inference?
I have a lot of duplicate images. I need to make a selection to reduce the amount of images the Mask RCNN model will perform inference on.
In every collection of duplicates, the images slightly differ....
0
votes
0
answers
27
views
How to balance classes for YOLO?
The problem I am having is that to my understanding we need to annotate all objects of all classes on the images we want to train (or fine tune) our YOLO on. This is because YOLO compares labeled ...
- 1,240
1
vote
0
answers
32
views
How does one deal with images that are too large to fit in the GPU memory for doing ML image analysis?
How does one deal with images that are too large to fit in the GPU memory for doing ML image analysis?
I am interested in detecting small structures on images which are themselves many GB in size. ...
- 111
0
votes
0
answers
12
views
How to generate multi-class segmentation masks for grapevine(plant) having the image (mostly white background) and the skeleton of the plant?
I have images of plants (grapevine) on mostly white background.
I have the skeleton of the plant in graph representation, where each edge has a label - the category of the edge. The categories ...
- 101
1
vote
0
answers
20
views
How to prepare audio data for deep learning?
Audio data is typically an array with the waveform represented by values from -1 to 1. There are two issues with that:
if all values are inverted, e.g. -1 becomes 1 and 1 becomes -1, the audio doesn'...
- 11
0
votes
0
answers
13
views
How would I encode a variable-length array to use an SVM?
I'm working on some image processing, and I have a list of contours
(it's essentially a list [or array] of (X, Y) coordinate pairs). These vary in length, depending on the size of the found contours.
...
- 101
1
vote
1
answer
33
views
Are there any advantages of encoding an image as a graph to use in Graph Convolutional Networks?
I have seen this encoding of an image as a graph:
The set of the nodes $V$ is the set of pixels. If the image is of size $10\times10$, then we have $10\cdot10=100$ pixels.
Each node has a length 3 ...
0
votes
2
answers
36
views
Would it be a problem to use non-squared images for a CNN model?
I want to create a music sheet scanner using CNN Model and the images I am using are not squared and, if I make them squared, important data will be lost and it might confuse the model.
Is it ok to ...
0
votes
0
answers
24
views
What is a good strategy for breaking up content into prompts and completions for OpenAI fine tuning?
I want to train a fine-tuned openai model to know more about specific Judo throws and training methodologies. I have a bunch of documents I have written on Judo throws that I would like to use for ...
- 101
0
votes
0
answers
23
views
How to organize data for training a neural network to raytrace images
I am trying to train a neural network to go from a series of numbers(data about a 3D scene, such as camera position, sphere position, radius, and color, etc) to a raytraced image. I can generate as ...
- 1
0
votes
0
answers
13
views
What are the best ways to preprocess landmarks to train a Neural Network?
I'd like to know, generally speaking, which are the most useful ways to preprocess landmarks to use as training set for a Deep Neural Network. Since they're put in a 3D space, would that be enough ...
- 1
9
votes
2
answers
1k
views
How can I encode angle data to train neural networks?
I am training a neural network where the target data is a vector of angles in radians (between $0$ and $2\pi$).
I am looking for study material on how to encode this data.
Can you supply me with a ...
- 291
0
votes
0
answers
20
views
bad prediction when having noise on the data: LSTM time-series regression
I want to predict the force plate using a smart insole using the LSTM model for time series prediction. the data on the force plate has positive and negative values (I think the resulting positive ...
2
votes
0
answers
21
views
How to decide which column has more weightage to output
As per Image we can see Column_A value is directly proportional to output,
While Change in value of Column_B has no effects in output.
So basically I want to know is there any algorithm where I can ...
- 21
0
votes
0
answers
34
views
How to handle missing data for an ordinal feature variable for a deep learning model?
BACKGROUND: I am developing a deep learning model in which one feature variable (out of many) is the grade of cancer, an ordinal variable. Below is a breakdown of my data by grade:
...
- 882
0
votes
0
answers
15
views
When training an LSTM, should you pad your dataset so the sequence length is static, or should it be variable?
I am putting together an LSTM network using visual basic. It's more of a learning exercise really, but it's also the only programming language I have access too at work.
I am unsure of how to prepare ...
- 323
0
votes
1
answer
60
views
How to encode categorical data for a convolutional model?
Is there a way to encode categorical nominal (no ordered) data to be used in CNN models?
Let's say I need to create a 1D CNN model for categorization of time series but the values are not measurements,...
- 1,240
0
votes
0
answers
31
views
How to filter point data based on pattern
I have a set of points represented as (x, y), and wish to separate out chunks/clusters of them which represent a ball trajectory.
For example, in the two sample images below I wish to find out cluster ...
- 76
1
vote
1
answer
151
views
Feature Engineering on transactional dataset clustering
I have a data set with transactions details from different business (roughly 1 thousand business entities). Each row is a transaction. The structure of the dataset is as follows:
client_id
Sex
Age
...
0
votes
0
answers
30
views
Retrieving data from a table by matching ID values
I have two columns [Uid, Con_id]; I need to fetch Uid id's which are matched with some Con_id's [From 606 to 615].
For example, all user_id's in each con_id from 606 to 615. But each Uid should match ...
0
votes
1
answer
164
views
combine two features in dataset?
I have a data set containing the number of security gaps and the level of that gap for a specific website.
Now suppose I have 2 features in this data set, the first feature is the number of a ...
0
votes
1
answer
24
views
Adding several variables that could be important but can introduce overfitting
Sopose a productivity dataset, where day of the week and months day number are important. I'm thinking to encode these with a one-hot encoding.
But if you have few years of data, that features might ...
- 121
0
votes
1
answer
25
views
How to encode both sentences and categorical data?
I have a DataFrame that contains several columns where some columns contain single words that can be category encoded since I know how many of them are there in total. However one column is an actual ...
- 103
0
votes
0
answers
35
views
Raw Audio Data Learning with CNN: Use zero-centered Input with ReLu?
I am playing around with Conv-Nets on raw audio data. Found some papers that outline different architectures but did not find a lot about the data preprocessing.
Can I use a zero-centred input for a ...
- 1
0
votes
0
answers
12
views
Where have I gone wrong? Data Preprocessing and cross validation
I have the following steps:
Fill in missing values - 'mean' for continuous, '10' for discrete columns (discrete columns go up to 0,1,2,3,4,5,6)
Feature selection (correlation based, whole dataset)
...
0
votes
1
answer
66
views
How to detect and deal with data distribution drift/change?
I'm working on a problem in ML to assess the performance of multiple vendors. I have a set of features in my dataset, and it appears each vendor is characterized by its own distribution. This is my ...
0
votes
1
answer
139
views
Can I use discrete data in the same model as continuous data?
In my dataset, I have some data that is continuous - eg. Age and BMI. I also have some data that is discrete- for example, occupation is labelled as 1 ="Homemaker"
2="Working"
3=&...
0
votes
0
answers
66
views
Python code for Background label removal from Biomedical images
I am doing research in Biomedical Image processing and Deep Learning using Python language. I have labels in the background (as highlighted in yellow ) and Pectoral muscle ( as highlighted in red) of ...
- 1
0
votes
2
answers
33
views
Can I shuffle data for delivery duration forecast problem?
I'm new to ML and trying to write a solution to a food delivery duration time problem (so called lead time). I used algorithms such as random forest and gradient boosting which gave OK results but not ...
1
vote
0
answers
12
views
how to manage the impact of Covid on building a machine learning model
I need your suggestions for using historical data to build a machine learning model for analyzing the market and build an AI model(tree based model/random forest or regression analysis) for setting ...
- 111
1
vote
2
answers
94
views
What makes a 'good' dataset
for the usage of ML technologies, having a appropriate dataset is arguably the first and fundamental step one has to tackle by either aquiring a dataset from external sources or creating their own.
...
- 11
0
votes
0
answers
45
views
How to Extract Specific Information From Web Pages Containing Job Postings?
I am currently working on my first research project, which is to implement machine learning based web scraping in the e-recruitement environment. Currently, I am building the dataset that needs to be ...
- 11
0
votes
0
answers
25
views
How to train DINO on images with varying length (excluding padding options)?
I would like to train DINO (Emerging Properties in Self-Supervised Vision Transformers) on spectrograms of different size (specifically: different number of time bins and same number of frequency bins)...
- 1
0
votes
1
answer
61
views
Is there a way to improve the low-quality data?
I'm on a robotics team and we've been tasked to write a program to differentiate between a live and dead fish. We've been given ~15 minutes of training footage and it's absolutely terrible. It's low ...
- 15
2
votes
3
answers
602
views
Do I need to normalize all state-space variables? If so, how?
I am playing around with a DRL agent in a stock-trading environment.
I have normalized all the external input data (the features that my agent will use). However, what about characteristics that don't ...
- 312
0
votes
1
answer
370
views
How should you reshape data before feeding it to LSTM layers?
I was curious if anyone had any advice on how to reshape data for a recurrent neural network. What I've been doing is array.reshape(len(X_train), # of points in time, # of features)
And then in the ...
-1
votes
1
answer
37
views
Is this the right approach to preprocessing data for artificial neural-networks? [closed]
I recently participated in a competitive "hackathon" with the problem being binary classification of overall satisfaction for travelers. The dataset mostly consisted of survey questions and ...
0
votes
0
answers
28
views
Using an RNN for predicting columns of characters
I'm making an RNN using pytorch to learn from columns of tiles (each tile represented by a text character) and predict the next column of tiles. The training sequences are from maps of level data ...
- 1
0
votes
2
answers
66
views
How to represent multi-label colours in one-hot encoding?
Say I want to predict the price of a gemstone based on its colour.
I have two options:
averaging over its colour on an RGB scale, or
using its textual description.
If I was to choose the latter, how ...
- 37
5
votes
1
answer
148
views
Does the term "data augmentation" imply increasing the training dataset?
I have a manuscript that has been reviewed and one of the reviewers commented on my use of the term " data augmentation", saying that it might not be the appropriate term in my case (...
- 181
2
votes
3
answers
74
views
Why my classification results are correlated with the proportionality of my data?
I'm facing a problem. I'm working on mixed data model with NN (MLP & Word Embedding). My results are not pretty good. And I observed that the proportionality of my data are corelated with my ...
0
votes
1
answer
96
views
Normalizing float prices with movements up to a factor of 100
I have a bunch of arbitrary float numbers (asset prices), that I have to feed into a neural network.
In the data set: values are between 1E-10 and 1E6
In a single sample: values may differ by a ...
0
votes
1
answer
40
views
How can my RNN get way better results than my ANN [closed]
So, I'm using the same dataset in both models but my RNN gets a 95% accuracy and my ANN gets 52%.
It is a time series, binary classification problem, and I know that RNN is better than ANN for time ...
1
vote
0
answers
234
views
How do I deal with a dataset of Images with variable sizes (width and height) when doing Image Classification?
I have a dataset in which the images which don't have the same width and height. How do I perform Image Classification with such images? I am trying as much as possible to steer away from image ...
- 11
0
votes
0
answers
31
views
How can the agent be defined in a reinforcement learning problem with a tabular dataset as the environment?
Let's assume we need to train an RL model that drops duplicates in a tabular dataset? The actions should probably defined as drop or do nothing.
But what should be the agent itself then? To me, it ...
- 1
0
votes
1
answer
48
views
how to detect ouliers in audio dataset?
I'm currently working on an audio classification project using CNNs. The problem is I'm having trouble training my CNN. I doubt if there are outliers in my dataset but I don't know how to detect ...
0
votes
1
answer
57
views
Which generalization of standard deviation to use for multidimensional input normalization
For machine learning tasks, it's common to normalize input data by subtracting the mean $\mu$ and dividing by the standard deviation $\sigma$ of the dataset:
$$\hat{x_i} = \frac{x_i - \mu}{\sigma}$$
—...
- 103
1
vote
0
answers
29
views
Transforming Moving Position Data to an Inputvector for Neural Networks
Imagine a car is driving on a long street (= x-axis). The car can go in both directions and it will arbitrarily change its direction.
I'm trying to formulate an Inputvector to tell a neural Network ...
- 11