Questions tagged [data-preprocessing]
For questions related to the concept of data pre-processing, which includes, for example, cleaning, instance selection, normalization, transformation, feature extraction or selection.
169
questions
0
votes
0
answers
18
views
Same prediction result with little probabilities change
I build a job prediction system leveraging data scrapped from LinkedIn with Random Forest and compared to XGBoost. XGBoost was used due to high accuracy after training.
When I made a prediction, I ...
- 1
0
votes
0
answers
6
views
Dimensions and file sizes of AI types (img/txt/sound) that work commercially and projections of future AI based on dimensionality and file simplicity?
Is it accurate that current AI breakthroughs are proportional to low dimensional complexity of datasets and small data throughput?
Image and text file renders are 100's of kilobytes, and they are ...
- 141
0
votes
0
answers
19
views
Can't get a correct accuracy on tabular data using deep learning
This is my first message here, and I would like to seek some assistance !
I have a technical test for a job that I really want, and I have 10 days to complete it. I've attempted to work on it, but I'm ...
0
votes
0
answers
7
views
Using Variable Text Features in Machine Learning Model
I am building a classification model using deidentified patient data with ICD-10 codes as inputs. Each code is a string and represents a diagnosis, and these follow the pattern of 1 letter, followed ...
- 101
-2
votes
2
answers
39
views
Can the multiclass classification be modified to binary classification for better accuracy [closed]
I've achieved good accuracy in binary classification problems, but I'm facing challenges when it comes to multi-class classification tasks.
While tackling a natural language processing challenge ...
0
votes
1
answer
28
views
Data out of scale
Imagine that you want to train a deep learning model for forecasting/transforming data and have a training dataset with endpoints 0 and 1000. Now, Imagine receiving a new dataset with endpoints 0 and ...
1
vote
1
answer
148
views
What causes my loss curve to consistently oscillate when training an LLM?
Why is my loss curve consistently oscillating? Every 50 steps it jumps back up. I'm assumming there's a bug in my data, since I'm using this colab notebook that shows a proper train/loss at the bottom....
- 11
0
votes
1
answer
38
views
When training a DNN model, how can I make some training data points more important than others?
In other words, is it possible to assign "weights" to data points during model training? Is there a standard technique for it?
It seems like it the math would be straightforward enough for ...
- 103
0
votes
0
answers
18
views
Handling Feature Selection Discrepancy in Image Classification Model
I have developed an image classification model that categorizes images into two classes (we'll say good and bad for the sake of example) based on a set of tags. To improve the model's performance, I ...
0
votes
1
answer
77
views
Does data preprocessing is necessary and important in deep learning?
I really wonder about data preprocessing is really necessary and important in deep learning.
It's really hard to say clearly about difference between Machine learning and deep learning. In definetion, ...
- 27
0
votes
0
answers
26
views
How to identify location from a given input in a semantic way
I have a sample use case, where user will give us a xls file with some location data. Where may be location information is given in different ways , sometime its IATA code, sometimes its standard ...
- 1
3
votes
1
answer
141
views
How are sentences numerically encoded before passing them to neural networks?
I'm trying to understand NLP, how sentences can be used as input output in neural network architecture. As we know ANN is only compatible with number data. That's mean the sentences must be convert to ...
0
votes
1
answer
24
views
Should I apply a min-max scale (range 0 to 1) before applying the normalisation or should I apply the z-score normalisation directly?
I want to implement a neural network in Pytorch for medical image segmentation. I should normalise my data.
Should I apply a min-max scale (range 0 to 1) before applying the normalisation or should I ...
- 101
0
votes
1
answer
643
views
Which preprocessing is the correct way to forecast time-series data using LSTM?
I just started to study time-series forecasting using RNN.
I have a few months of time series data that was an hour unit.
The data is a kind of percentage value of my little experiment and I would ...
- 103
0
votes
0
answers
7
views
Should I downsample because of overrepresentation of geographic locations in time series data
I am in the start of working with a project where I am hoping to be able to classify activities based time series data. I have historic data; lat/long/speed/(..) as well as the activity.
The challenge ...
- 101
2
votes
1
answer
63
views
AI to understand what problem my computer has, for instance by looking at the logs which are too big for me
Short version
I have a computer which doesn't work properly but I can't figure out what's wrong. I've tried to install other operating systems, run some system tests and look at the system logs, but ...
- 121
3
votes
1
answer
628
views
Process 2TB worth of conversational data hoarded over 40 years. How can I pass this into GPT to ask questions about it?
I'm still very new to this stuff. I have close to 2TB worth of data hoarded from IRC chats to everyday chats with friends and family.
But is there a way to pass in this much data into GPT to ask ...
- 139
0
votes
0
answers
6
views
Comparing Quality of Raw Data from Various Hardware Setups
My background is not in ML so please correct any gaps in my question.
I am working on an application trying to discriminate across multiple sounds (eg. bark, glass break, keys jingle etc.). There are ...
- 101
0
votes
0
answers
38
views
True way to pad input image for Transformer OCR model?
I have a Transformer model for OCR problem. I would like to resize input image while keeping image ratio. How should I do? (I prefer 1 but I have not tried yet)
Keep width fit to new width, scale ...
1
vote
2
answers
73
views
Is manual binding output to input also an AI?
I know AI is primarly training a machine by samples of input-output in order it would learn itself about relations between the input and the output.
What if I manually add the relations? Is that still ...
- 13
-1
votes
2
answers
35
views
Linking preprocessing with models [closed]
We have been working on a deep learning problem for a few iterations now. We've been tweaking the preprocessing as we go. We've also been training models as we go.
The people using the models for ...
- 125
0
votes
0
answers
33
views
What computer vision techniques can help me select the best image for object detection inference?
I have a lot of duplicate images. I need to make a selection to reduce the amount of images the Mask RCNN model will perform inference on.
In every collection of duplicates, the images slightly differ....
0
votes
0
answers
147
views
How to balance classes for YOLO?
The problem I am having is that to my understanding we need to annotate all objects of all classes on the images we want to train (or fine tune) our YOLO on. This is because YOLO compares labeled ...
- 1,240
2
votes
0
answers
73
views
How does one deal with images that are too large to fit in the GPU memory for doing ML image analysis?
How does one deal with images that are too large to fit in the GPU memory for doing ML image analysis?
I am interested in detecting small structures on images which are themselves many GB in size. ...
- 121
0
votes
0
answers
39
views
How to generate multi-class segmentation masks for grapevine(plant) having the image (mostly white background) and the skeleton of the plant?
I have images of plants (grapevine) on mostly white background.
I have the skeleton of the plant in graph representation, where each edge has a label - the category of the edge. The categories ...
- 101
2
votes
0
answers
31
views
How to prepare audio data for deep learning?
Audio data is typically an array with the waveform represented by values from -1 to 1. There are two issues with that:
if all values are inverted, e.g. -1 becomes 1 and 1 becomes -1, the audio doesn'...
- 21
0
votes
0
answers
23
views
How would I encode a variable-length array to use an SVM?
I'm working on some image processing, and I have a list of contours
(it's essentially a list [or array] of (X, Y) coordinate pairs). These vary in length, depending on the size of the found contours.
...
- 101
1
vote
1
answer
110
views
Are there any advantages of encoding an image as a graph to use in Graph Convolutional Networks?
I have seen this encoding of an image as a graph:
The set of the nodes $V$ is the set of pixels. If the image is of size $10\times10$, then we have $10\cdot10=100$ pixels.
Each node has a length 3 ...
0
votes
2
answers
85
views
Would it be a problem to use non-squared images for a CNN model?
I want to create a music sheet scanner using CNN Model and the images I am using are not squared and, if I make them squared, important data will be lost and it might confuse the model.
Is it ok to ...
0
votes
0
answers
43
views
What is a good strategy for breaking up content into prompts and completions for OpenAI fine tuning?
I want to train a fine-tuned openai model to know more about specific Judo throws and training methodologies. I have a bunch of documents I have written on Judo throws that I would like to use for ...
- 101
0
votes
0
answers
35
views
How to organize data for training a neural network to raytrace images
I am trying to train a neural network to go from a series of numbers(data about a 3D scene, such as camera position, sphere position, radius, and color, etc) to a raytraced image. I can generate as ...
- 1
0
votes
0
answers
23
views
What are the best ways to preprocess landmarks to train a Neural Network?
I'd like to know, generally speaking, which are the most useful ways to preprocess landmarks to use as training set for a Deep Neural Network. Since they're put in a 3D space, would that be enough ...
- 1
10
votes
2
answers
2k
views
How can I encode angle data to train neural networks?
I am training a neural network where the target data is a vector of angles in radians (between $0$ and $2\pi$).
I am looking for study material on how to encode this data.
Can you supply me with a ...
- 299
1
vote
0
answers
28
views
bad prediction when having noise on the data: LSTM time-series regression
I want to predict the force plate using a smart insole using the LSTM model for time series prediction. the data on the force plate has positive and negative values (I think the resulting positive ...
- 11
2
votes
0
answers
23
views
How to decide which column has more weightage to output
As per Image we can see Column_A value is directly proportional to output,
While Change in value of Column_B has no effects in output.
So basically I want to know is there any algorithm where I can ...
- 21
0
votes
0
answers
55
views
How to handle missing data for an ordinal feature variable for a deep learning model?
BACKGROUND: I am developing a deep learning model in which one feature variable (out of many) is the grade of cancer, an ordinal variable. Below is a breakdown of my data by grade:
...
- 922
0
votes
1
answer
119
views
How to encode categorical data for a convolutional model?
Is there a way to encode categorical nominal (no ordered) data to be used in CNN models?
Let's say I need to create a 1D CNN model for categorization of time series but the values are not measurements,...
- 1,240
1
vote
1
answer
265
views
Feature Engineering on transactional dataset clustering
I have a data set with transactions details from different business (roughly 1 thousand business entities). Each row is a transaction. The structure of the dataset is as follows:
client_id
Sex
Age
...
0
votes
1
answer
198
views
combine two features in dataset?
I have a data set containing the number of security gaps and the level of that gap for a specific website.
Now suppose I have 2 features in this data set, the first feature is the number of a ...
0
votes
1
answer
31
views
Adding several variables that could be important but can introduce overfitting
Sopose a productivity dataset, where day of the week and months day number are important. I'm thinking to encode these with a one-hot encoding.
But if you have few years of data, that features might ...
- 121
1
vote
1
answer
38
views
How to encode both sentences and categorical data?
I have a DataFrame that contains several columns where some columns contain single words that can be category encoded since I know how many of them are there in total. However one column is an actual ...
- 113
1
vote
1
answer
173
views
How to detect and deal with data distribution drift/change?
I'm working on a problem in ML to assess the performance of multiple vendors. I have a set of features in my dataset, and it appears each vendor is characterized by its own distribution. This is my ...
- 11
0
votes
1
answer
340
views
Can I use discrete data in the same model as continuous data?
In my dataset, I have some data that is continuous - eg. Age and BMI. I also have some data that is discrete- for example, occupation is labelled as 1 ="Homemaker"
2="Working"
3=&...
0
votes
2
answers
39
views
Can I shuffle data for delivery duration forecast problem?
I'm new to ML and trying to write a solution to a food delivery duration time problem (so called lead time). I used algorithms such as random forest and gradient boosting which gave OK results but not ...
- 11
1
vote
0
answers
12
views
how to manage the impact of Covid on building a machine learning model
I need your suggestions for using historical data to build a machine learning model for analyzing the market and build an AI model(tree based model/random forest or regression analysis) for setting ...
- 111
1
vote
2
answers
217
views
What makes a 'good' dataset
for the usage of ML technologies, having a appropriate dataset is arguably the first and fundamental step one has to tackle by either aquiring a dataset from external sources or creating their own.
...
- 11
0
votes
1
answer
67
views
Is there a way to improve the low-quality data?
I'm on a robotics team and we've been tasked to write a program to differentiate between a live and dead fish. We've been given ~15 minutes of training footage and it's absolutely terrible. It's low ...
user55176
2
votes
3
answers
1k
views
Do I need to normalize all state-space variables? If so, how?
I am playing around with a DRL agent in a stock-trading environment.
I have normalized all the external input data (the features that my agent will use). However, what about characteristics that don't ...
- 342
0
votes
1
answer
543
views
How should you reshape data before feeding it to LSTM layers?
I was curious if anyone had any advice on how to reshape data for a recurrent neural network. What I've been doing is array.reshape(len(X_train), # of points in time, # of features)
And then in the ...
-1
votes
1
answer
47
views
Is this the right approach to preprocessing data for artificial neural-networks? [closed]
I recently participated in a competitive "hackathon" with the problem being binary classification of overall satisfaction for travelers. The dataset mostly consisted of survey questions and ...
