Questions tagged [data-preprocessing]

For questions related to the concept of data pre-processing, which includes, for example, cleaning, instance selection, normalization, transformation, feature extraction or selection.

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

Same prediction result with little probabilities change

I build a job prediction system leveraging data scrapped from LinkedIn with Random Forest and compared to XGBoost. XGBoost was used due to high accuracy after training. When I made a prediction, I ...
ezaryf's user avatar
  • 1
0 votes
0 answers
6 views

Dimensions and file sizes of AI types (img/txt/sound) that work commercially and projections of future AI based on dimensionality and file simplicity?

Is it accurate that current AI breakthroughs are proportional to low dimensional complexity of datasets and small data throughput? Image and text file renders are 100's of kilobytes, and they are ...
bandybabboon's user avatar
0 votes
0 answers
19 views

Can't get a correct accuracy on tabular data using deep learning

This is my first message here, and I would like to seek some assistance ! I have a technical test for a job that I really want, and I have 10 days to complete it. I've attempted to work on it, but I'm ...
Enzo Durand's user avatar
0 votes
0 answers
7 views

Using Variable Text Features in Machine Learning Model

I am building a classification model using deidentified patient data with ICD-10 codes as inputs. Each code is a string and represents a diagnosis, and these follow the pattern of 1 letter, followed ...
Omnitragedy's user avatar
-2 votes
2 answers
39 views

Can the multiclass classification be modified to binary classification for better accuracy [closed]

I've achieved good accuracy in binary classification problems, but I'm facing challenges when it comes to multi-class classification tasks. While tackling a natural language processing challenge ...
Raghul Azhagaiah's user avatar
0 votes
1 answer
28 views

Data out of scale

Imagine that you want to train a deep learning model for forecasting/transforming data and have a training dataset with endpoints 0 and 1000. Now, Imagine receiving a new dataset with endpoints 0 and ...
Nathaldien's user avatar
1 vote
1 answer
148 views

What causes my loss curve to consistently oscillate when training an LLM?

Why is my loss curve consistently oscillating? Every 50 steps it jumps back up. I'm assumming there's a bug in my data, since I'm using this colab notebook that shows a proper train/loss at the bottom....
bmatzelle's user avatar
0 votes
1 answer
38 views

When training a DNN model, how can I make some training data points more important than others?

In other words, is it possible to assign "weights" to data points during model training? Is there a standard technique for it? It seems like it the math would be straightforward enough for ...
PlinyTheElder's user avatar
0 votes
0 answers
18 views

Handling Feature Selection Discrepancy in Image Classification Model

I have developed an image classification model that categorizes images into two classes (we'll say good and bad for the sake of example) based on a set of tags. To improve the model's performance, I ...
eszfgefr rgrer's user avatar
0 votes
1 answer
77 views

Does data preprocessing is necessary and important in deep learning?

I really wonder about data preprocessing is really necessary and important in deep learning. It's really hard to say clearly about difference between Machine learning and deep learning. In definetion, ...
Yang's user avatar
  • 27
0 votes
0 answers
26 views

How to identify location from a given input in a semantic way

I have a sample use case, where user will give us a xls file with some location data. Where may be location information is given in different ways , sometime its IATA code, sometimes its standard ...
Sujoy's user avatar
  • 1
3 votes
1 answer
141 views

How are sentences numerically encoded before passing them to neural networks?

I'm trying to understand NLP, how sentences can be used as input output in neural network architecture. As we know ANN is only compatible with number data. That's mean the sentences must be convert to ...
Muhammad Ikhwan Perwira's user avatar
0 votes
1 answer
24 views

Should I apply a min-max scale (range 0 to 1) before applying the normalisation or should I apply the z-score normalisation directly?

I want to implement a neural network in Pytorch for medical image segmentation. I should normalise my data. Should I apply a min-max scale (range 0 to 1) before applying the normalisation or should I ...
Janikas's user avatar
  • 101
0 votes
1 answer
643 views

Which preprocessing is the correct way to forecast time-series data using LSTM?

I just started to study time-series forecasting using RNN. I have a few months of time series data that was an hour unit. The data is a kind of percentage value of my little experiment and I would ...
orde.r's user avatar
  • 103
0 votes
0 answers
7 views

Should I downsample because of overrepresentation of geographic locations in time series data

I am in the start of working with a project where I am hoping to be able to classify activities based time series data. I have historic data; lat/long/speed/(..) as well as the activity. The challenge ...
bjornasm's user avatar
  • 101
2 votes
1 answer
63 views

AI to understand what problem my computer has, for instance by looking at the logs which are too big for me

Short version I have a computer which doesn't work properly but I can't figure out what's wrong. I've tried to install other operating systems, run some system tests and look at the system logs, but ...
peoro's user avatar
  • 121
3 votes
1 answer
628 views

Process 2TB worth of conversational data hoarded over 40 years. How can I pass this into GPT to ask questions about it?

I'm still very new to this stuff. I have close to 2TB worth of data hoarded from IRC chats to everyday chats with friends and family. But is there a way to pass in this much data into GPT to ask ...
Patoshi パトシ's user avatar
0 votes
0 answers
6 views

Comparing Quality of Raw Data from Various Hardware Setups

My background is not in ML so please correct any gaps in my question. I am working on an application trying to discriminate across multiple sounds (eg. bark, glass break, keys jingle etc.). There are ...
EasyOhm's user avatar
  • 101
0 votes
0 answers
38 views

True way to pad input image for Transformer OCR model?

I have a Transformer model for OCR problem. I would like to resize input image while keeping image ratio. How should I do? (I prefer 1 but I have not tried yet) Keep width fit to new width, scale ...
Rhaenys98's user avatar
1 vote
2 answers
73 views

Is manual binding output to input also an AI?

I know AI is primarly training a machine by samples of input-output in order it would learn itself about relations between the input and the output. What if I manually add the relations? Is that still ...
stkuser's user avatar
  • 13
-1 votes
2 answers
35 views

Linking preprocessing with models [closed]

We have been working on a deep learning problem for a few iterations now. We've been tweaking the preprocessing as we go. We've also been training models as we go. The people using the models for ...
Paul Reiners's user avatar
0 votes
0 answers
33 views

What computer vision techniques can help me select the best image for object detection inference?

I have a lot of duplicate images. I need to make a selection to reduce the amount of images the Mask RCNN model will perform inference on. In every collection of duplicates, the images slightly differ....
Nick De Wispelaere's user avatar
0 votes
0 answers
147 views

How to balance classes for YOLO?

The problem I am having is that to my understanding we need to annotate all objects of all classes on the images we want to train (or fine tune) our YOLO on. This is because YOLO compares labeled ...
GKozinski's user avatar
  • 1,240
2 votes
0 answers
73 views

How does one deal with images that are too large to fit in the GPU memory for doing ML image analysis?

How does one deal with images that are too large to fit in the GPU memory for doing ML image analysis? I am interested in detecting small structures on images which are themselves many GB in size. ...
Luca's user avatar
  • 121
0 votes
0 answers
39 views

How to generate multi-class segmentation masks for grapevine(plant) having the image (mostly white background) and the skeleton of the plant?

I have images of plants (grapevine) on mostly white background. I have the skeleton of the plant in graph representation, where each edge has a label - the category of the edge. The categories ...
Hidi Eric's user avatar
  • 101
2 votes
0 answers
31 views

How to prepare audio data for deep learning?

Audio data is typically an array with the waveform represented by values from -1 to 1. There are two issues with that: if all values are inverted, e.g. -1 becomes 1 and 1 becomes -1, the audio doesn'...
nikishev.'s user avatar
0 votes
0 answers
23 views

How would I encode a variable-length array to use an SVM?

I'm working on some image processing, and I have a list of contours (it's essentially a list [or array] of (X, Y) coordinate pairs). These vary in length, depending on the size of the found contours. ...
Kevin's user avatar
  • 101
1 vote
1 answer
110 views

Are there any advantages of encoding an image as a graph to use in Graph Convolutional Networks?

I have seen this encoding of an image as a graph: The set of the nodes $V$ is the set of pixels. If the image is of size $10\times10$, then we have $10\cdot10=100$ pixels. Each node has a length 3 ...
puradrogasincortar's user avatar
0 votes
2 answers
85 views

Would it be a problem to use non-squared images for a CNN model?

I want to create a music sheet scanner using CNN Model and the images I am using are not squared and, if I make them squared, important data will be lost and it might confuse the model. Is it ok to ...
CelticusAndy's user avatar
0 votes
0 answers
43 views

What is a good strategy for breaking up content into prompts and completions for OpenAI fine tuning?

I want to train a fine-tuned openai model to know more about specific Judo throws and training methodologies. I have a bunch of documents I have written on Judo throws that I would like to use for ...
Casey Jordan's user avatar
0 votes
0 answers
35 views

How to organize data for training a neural network to raytrace images

I am trying to train a neural network to go from a series of numbers(data about a 3D scene, such as camera position, sphere position, radius, and color, etc) to a raytraced image. I can generate as ...
Fahd's user avatar
  • 1
0 votes
0 answers
23 views

What are the best ways to preprocess landmarks to train a Neural Network?

I'd like to know, generally speaking, which are the most useful ways to preprocess landmarks to use as training set for a Deep Neural Network. Since they're put in a 3D space, would that be enough ...
Leno's user avatar
  • 1
10 votes
2 answers
2k views

How can I encode angle data to train neural networks?

I am training a neural network where the target data is a vector of angles in radians (between $0$ and $2\pi$). I am looking for study material on how to encode this data. Can you supply me with a ...
user366312's user avatar
1 vote
0 answers
28 views

bad prediction when having noise on the data: LSTM time-series regression

I want to predict the force plate using a smart insole using the LSTM model for time series prediction. the data on the force plate has positive and negative values (I think the resulting positive ...
stack offer's user avatar
2 votes
0 answers
23 views

How to decide which column has more weightage to output

As per Image we can see Column_A value is directly proportional to output, While Change in value of Column_B has no effects in output. So basically I want to know is there any algorithm where I can ...
Marques's user avatar
  • 21
0 votes
0 answers
55 views

How to handle missing data for an ordinal feature variable for a deep learning model?

BACKGROUND: I am developing a deep learning model in which one feature variable (out of many) is the grade of cancer, an ordinal variable. Below is a breakdown of my data by grade: ...
Snehal Patel's user avatar
0 votes
1 answer
119 views

How to encode categorical data for a convolutional model?

Is there a way to encode categorical nominal (no ordered) data to be used in CNN models? Let's say I need to create a 1D CNN model for categorization of time series but the values are not measurements,...
GKozinski's user avatar
  • 1,240
1 vote
1 answer
265 views

Feature Engineering on transactional dataset clustering

I have a data set with transactions details from different business (roughly 1 thousand business entities). Each row is a transaction. The structure of the dataset is as follows: client_id Sex Age ...
Juan Ignacio Rojo's user avatar
0 votes
1 answer
198 views

combine two features in dataset?

I have a data set containing the number of security gaps and the level of that gap for a specific website. Now suppose I have 2 features in this data set, the first feature is the number of a ...
Issa Mansour's user avatar
0 votes
1 answer
31 views

Adding several variables that could be important but can introduce overfitting

Sopose a productivity dataset, where day of the week and months day number are important. I'm thinking to encode these with a one-hot encoding. But if you have few years of data, that features might ...
Mquinteiro's user avatar
1 vote
1 answer
38 views

How to encode both sentences and categorical data?

I have a DataFrame that contains several columns where some columns contain single words that can be category encoded since I know how many of them are there in total. However one column is an actual ...
MilTom's user avatar
  • 113
1 vote
1 answer
173 views

How to detect and deal with data distribution drift/change?

I'm working on a problem in ML to assess the performance of multiple vendors. I have a set of features in my dataset, and it appears each vendor is characterized by its own distribution. This is my ...
John Coles's user avatar
0 votes
1 answer
340 views

Can I use discrete data in the same model as continuous data?

In my dataset, I have some data that is continuous - eg. Age and BMI. I also have some data that is discrete- for example, occupation is labelled as 1 ="Homemaker" 2="Working" 3=&...
helpwithkeras's user avatar
0 votes
2 answers
39 views

Can I shuffle data for delivery duration forecast problem?

I'm new to ML and trying to write a solution to a food delivery duration time problem (so called lead time). I used algorithms such as random forest and gradient boosting which gave OK results but not ...
John Coles's user avatar
1 vote
0 answers
12 views

how to manage the impact of Covid on building a machine learning model

I need your suggestions for using historical data to build a machine learning model for analyzing the market and build an AI model(tree based model/random forest or regression analysis) for setting ...
Mahsa's user avatar
  • 111
1 vote
2 answers
217 views

What makes a 'good' dataset

for the usage of ML technologies, having a appropriate dataset is arguably the first and fundamental step one has to tackle by either aquiring a dataset from external sources or creating their own. ...
JakobS's user avatar
  • 11
0 votes
1 answer
67 views

Is there a way to improve the low-quality data?

I'm on a robotics team and we've been tasked to write a program to differentiate between a live and dead fish. We've been given ~15 minutes of training footage and it's absolutely terrible. It's low ...
user avatar
2 votes
3 answers
1k views

Do I need to normalize all state-space variables? If so, how?

I am playing around with a DRL agent in a stock-trading environment. I have normalized all the external input data (the features that my agent will use). However, what about characteristics that don't ...
Vladimir Belik's user avatar
0 votes
1 answer
543 views

How should you reshape data before feeding it to LSTM layers?

I was curious if anyone had any advice on how to reshape data for a recurrent neural network. What I've been doing is array.reshape(len(X_train), # of points in time, # of features) And then in the ...
Christopher Palmer's user avatar
-1 votes
1 answer
47 views

Is this the right approach to preprocessing data for artificial neural-networks? [closed]

I recently participated in a competitive "hackathon" with the problem being binary classification of overall satisfaction for travelers. The dataset mostly consisted of survey questions and ...
FIRE KhaN's user avatar