Skip to main content

Questions tagged [data-preprocessing]

For questions related to the concept of data pre-processing, which includes, for example, cleaning, instance selection, normalization, transformation, feature extraction or selection.

Filter by
Sorted by
Tagged with
0 votes
0 answers
26 views

Advice on Handling Variable Input Sizes in a Neural Network Model for Predicting Football Match Outcomes

I am developing a machine learning model aimed at predicting football match outcomes based on past team performances. The model incorporates data from the last 10 home games for the home team and the ...
Drk's user avatar
  • 1
1 vote
1 answer
54 views

How to evaluate the KL divergence between two distributions that may require sampling?

The KL divergence between two distributions is: \begin{equation} \int \mathbf{p}(x;\theta_{1}) \; log \frac{\mathbf{p}(x;\theta_{1})}{\mathbf{p}(x;\theta_{2})} \nu(dx) \\ \end{equation} If the ...
xiaolingxiao's user avatar
0 votes
0 answers
17 views

How to construct source padding mask for embedded audio?

I'm attempting a music transcription task - similar to speech recognition but with music and notes (string representations) instead of speech audio and sentences. The model consists of a CNN audio ...
jy99's user avatar
  • 1
0 votes
1 answer
29 views

Can I scale subsets of my dataset independently to handle different feature ranges?

I am currently making an LSTM model to predict the change in value of a stock for each day. My dataset consists of 4 years of data for 30 different stocks along with some financial metrics such as RSI,...
Daniel goldney's user avatar
0 votes
0 answers
23 views

Validating bundles of XMLs with inconsistent structure

The problem: We have a large number of XML bundles, and each bundle needs to meet certain criteria to be considered valid; specifically, certain values (or types of values) should belong to certain ...
SpaghettiM4ster's user avatar
2 votes
1 answer
70 views

Is it a requirement/recommendation to normalize my inputs into [0,1] range?

All features of my input dataset, which is going to be used for training a simple multi-layered neural network, are in range $[-1,+1]$ and the output of $NN$ is a single number again in range $[-1,+1]$...
Bikay's user avatar
  • 23
0 votes
0 answers
12 views

3-way hold-out for picking between different ML models

I have a dataset which is a time series. Before jumping on more heavy models such as LSTM, I wanted to test out the performance of linear models. I have currently a 80/20 split between training and ...
xingern's user avatar
0 votes
0 answers
25 views

Realtime Inference of Stateful Variable-Length Time Series LSTM - Getting around Sequence Length vs Inference Frequency Tradeoff

I am trying to train a neural net for low-latency signal filtering; I've been recommended to use a stateful LSTM architecture for this task, however, having corrected some , it seems to me that trying ...
stellarpower's user avatar
0 votes
0 answers
22 views

Same prediction result with little probabilities change

I built a job prediction system leveraging data scrapped from LinkedIn with Random Forest and compared it to XGBoost. XGBoost was used due to its high accuracy after training. When I made a prediction,...
ezaryf's user avatar
  • 1
0 votes
0 answers
10 views

Dimensions and file sizes of AI types (img/txt/sound) that work commercially and projections of future AI based on dimensionality and file simplicity?

Is it accurate that current AI breakthroughs are proportional to low dimensional complexity of datasets and small data throughput? Image and text file renders are 100's of kilobytes, and they are ...
bandybabboon's user avatar
0 votes
0 answers
26 views

Can't get a correct accuracy on tabular data using deep learning

This is my first message here, and I would like to seek some assistance ! I have a technical test for a job that I really want, and I have 10 days to complete it. I've attempted to work on it, but I'm ...
Enzo Durand's user avatar
0 votes
0 answers
10 views

Using Variable Text Features in Machine Learning Model

I am building a classification model using deidentified patient data with ICD-10 codes as inputs. Each code is a string and represents a diagnosis, and these follow the pattern of 1 letter, followed ...
Omnitragedy's user avatar
-2 votes
2 answers
111 views

Can the multiclass classification be modified to binary classification for better accuracy [closed]

I've achieved good accuracy in binary classification problems, but I'm facing challenges when it comes to multi-class classification tasks. While tackling a natural language processing challenge ...
Raghul Azhagaiah's user avatar
0 votes
1 answer
37 views

Data out of scale

Imagine that you want to train a deep learning model for forecasting/transforming data and have a training dataset with endpoints 0 and 1000. Now, Imagine receiving a new dataset with endpoints 0 and ...
Nathaldien's user avatar
1 vote
1 answer
485 views

What causes my loss curve to consistently oscillate when training an LLM?

Why is my loss curve consistently oscillating? Every 50 steps it jumps back up. I'm assumming there's a bug in my data, since I'm using this colab notebook that shows a proper train/loss at the bottom....
bmatzelle's user avatar
0 votes
1 answer
45 views

When training a DNN model, how can I make some training data points more important than others?

In other words, is it possible to assign "weights" to data points during model training? Is there a standard technique for it? It seems like it the math would be straightforward enough for ...
PlinyTheElder's user avatar
0 votes
0 answers
19 views

Handling Feature Selection Discrepancy in Image Classification Model

I have developed an image classification model that categorizes images into two classes (we'll say good and bad for the sake of example) based on a set of tags. To improve the model's performance, I ...
eszfgefr rgrer's user avatar
0 votes
1 answer
144 views

Does data preprocessing is necessary and important in deep learning?

I really wonder about data preprocessing is really necessary and important in deep learning. It's really hard to say clearly about difference between Machine learning and deep learning. In definetion, ...
Yang's user avatar
  • 57
0 votes
0 answers
28 views

How to identify location from a given input in a semantic way

I have a sample use case, where user will give us a xls file with some location data. Where may be location information is given in different ways , sometime its IATA code, sometimes its standard ...
Sujoy's user avatar
  • 1
3 votes
1 answer
207 views

How are sentences numerically encoded before passing them to neural networks?

I'm trying to understand NLP, how sentences can be used as input output in neural network architecture. As we know ANN is only compatible with number data. That's mean the sentences must be convert to ...
Muhammad Ikhwan Perwira's user avatar
0 votes
1 answer
28 views

Should I apply a min-max scale (range 0 to 1) before applying the normalisation or should I apply the z-score normalisation directly?

I want to implement a neural network in Pytorch for medical image segmentation. I should normalise my data. Should I apply a min-max scale (range 0 to 1) before applying the normalisation or should I ...
Janikas's user avatar
  • 101
0 votes
1 answer
2k views

Which preprocessing is the correct way to forecast time-series data using LSTM?

I just started to study time-series forecasting using RNN. I have a few months of time series data that was an hour unit. The data is a kind of percentage value of my little experiment and I would ...
orde.r's user avatar
  • 103
2 votes
1 answer
81 views

AI to understand what problem my computer has, for instance by looking at the logs which are too big for me

Short version I have a computer which doesn't work properly but I can't figure out what's wrong. I've tried to install other operating systems, run some system tests and look at the system logs, but ...
peoro's user avatar
  • 121
3 votes
1 answer
633 views

Process 2TB worth of conversational data hoarded over 40 years. How can I pass this into GPT to ask questions about it?

I'm still very new to this stuff. I have close to 2TB worth of data hoarded from IRC chats to everyday chats with friends and family. But is there a way to pass in this much data into GPT to ask ...
Patoshi パトシ's user avatar
1 vote
2 answers
83 views

Is manual binding output to input also an AI?

I know AI is primarly training a machine by samples of input-output in order it would learn itself about relations between the input and the output. What if I manually add the relations? Is that still ...
stkuser's user avatar
  • 13
-1 votes
2 answers
38 views

Linking preprocessing with models [closed]

We have been working on a deep learning problem for a few iterations now. We've been tweaking the preprocessing as we go. We've also been training models as we go. The people using the models for ...
Paul Reiners's user avatar
0 votes
0 answers
41 views

What computer vision techniques can help me select the best image for object detection inference?

I have a lot of duplicate images. I need to make a selection to reduce the amount of images the Mask RCNN model will perform inference on. In every collection of duplicates, the images slightly differ....
Nick De Wispelaere's user avatar
2 votes
0 answers
87 views

How does one deal with images that are too large to fit in the GPU memory for doing ML image analysis?

How does one deal with images that are too large to fit in the GPU memory for doing ML image analysis? I am interested in detecting small structures on images which are themselves many GB in size. ...
Luca's user avatar
  • 121
2 votes
0 answers
44 views

How to prepare audio data for deep learning?

Audio data is typically an array with the waveform represented by values from -1 to 1. There are two issues with that: if all values are inverted, e.g. -1 becomes 1 and 1 becomes -1, the audio doesn'...
Ford F150 Gaming's user avatar
1 vote
1 answer
191 views

Are there any advantages of encoding an image as a graph to use in Graph Convolutional Networks?

I have seen this encoding of an image as a graph: The set of the nodes $V$ is the set of pixels. If the image is of size $10\times10$, then we have $10\cdot10=100$ pixels. Each node has a length 3 ...
puradrogasincortar's user avatar
0 votes
2 answers
202 views

Would it be a problem to use non-squared images for a CNN model?

I want to create a music sheet scanner using CNN Model and the images I am using are not squared and, if I make them squared, important data will be lost and it might confuse the model. Is it ok to ...
CelticusAndy's user avatar
10 votes
2 answers
2k views

How can I encode angle data to train neural networks?

I am training a neural network where the target data is a vector of angles in radians (between $0$ and $2\pi$). I am looking for study material on how to encode this data. Can you supply me with a ...
user366312's user avatar
1 vote
0 answers
31 views

bad prediction when having noise on the data: LSTM time-series regression

I want to predict the force plate using a smart insole using the LSTM model for time series prediction. the data on the force plate has positive and negative values (I think the resulting positive ...
stack offer's user avatar
2 votes
0 answers
26 views

How to decide which column has more weightage to output

As per Image we can see Column_A value is directly proportional to output, While Change in value of Column_B has no effects in output. So basically I want to know is there any algorithm where I can ...
Marques's user avatar
  • 21
0 votes
0 answers
64 views

How to handle missing data for an ordinal feature variable for a deep learning model?

BACKGROUND: I am developing a deep learning model in which one feature variable (out of many) is the grade of cancer, an ordinal variable. Below is a breakdown of my data by grade: ...
Snehal Patel's user avatar
0 votes
1 answer
159 views

How to encode categorical data for a convolutional model?

Is there a way to encode categorical nominal (no ordered) data to be used in CNN models? Let's say I need to create a 1D CNN model for categorization of time series but the values are not measurements,...
GKozinski's user avatar
  • 1,270
1 vote
1 answer
352 views

Feature Engineering on transactional dataset clustering

I have a data set with transactions details from different business (roughly 1 thousand business entities). Each row is a transaction. The structure of the dataset is as follows: client_id Sex Age ...
Juan Ignacio Rojo's user avatar
0 votes
1 answer
226 views

combine two features in dataset?

I have a data set containing the number of security gaps and the level of that gap for a specific website. Now suppose I have 2 features in this data set, the first feature is the number of a ...
Issa Mansour's user avatar
0 votes
1 answer
36 views

Adding several variables that could be important but can introduce overfitting

Sopose a productivity dataset, where day of the week and months day number are important. I'm thinking to encode these with a one-hot encoding. But if you have few years of data, that features might ...
Mquinteiro's user avatar
1 vote
1 answer
42 views

How to encode both sentences and categorical data?

I have a DataFrame that contains several columns where some columns contain single words that can be category encoded since I know how many of them are there in total. However one column is an actual ...
MilTom's user avatar
  • 113
1 vote
1 answer
212 views

How to detect and deal with data distribution drift/change?

I'm working on a problem in ML to assess the performance of multiple vendors. I have a set of features in my dataset, and it appears each vendor is characterized by its own distribution. This is my ...
John Coles's user avatar
0 votes
1 answer
506 views

Can I use discrete data in the same model as continuous data?

In my dataset, I have some data that is continuous - eg. Age and BMI. I also have some data that is discrete- for example, occupation is labelled as 1 ="Homemaker" 2="Working" 3=&...
helpwithkeras's user avatar
0 votes
2 answers
48 views

Can I shuffle data for delivery duration forecast problem?

I'm new to ML and trying to write a solution to a food delivery duration time problem (so called lead time). I used algorithms such as random forest and gradient boosting which gave OK results but not ...
John Coles's user avatar
1 vote
0 answers
12 views

how to manage the impact of Covid on building a machine learning model

I need your suggestions for using historical data to build a machine learning model for analyzing the market and build an AI model(tree based model/random forest or regression analysis) for setting ...
Mahsa's user avatar
  • 111
1 vote
2 answers
278 views

What makes a 'good' dataset

for the usage of ML technologies, having a appropriate dataset is arguably the first and fundamental step one has to tackle by either aquiring a dataset from external sources or creating their own. ...
JakobS's user avatar
  • 11
0 votes
1 answer
75 views

Is there a way to improve the low-quality data?

I'm on a robotics team and we've been tasked to write a program to differentiate between a live and dead fish. We've been given ~15 minutes of training footage and it's absolutely terrible. It's low ...
user avatar
2 votes
3 answers
2k views

Do I need to normalize all state-space variables? If so, how?

I am playing around with a DRL agent in a stock-trading environment. I have normalized all the external input data (the features that my agent will use). However, what about characteristics that don't ...
Vladimir Belik's user avatar
0 votes
1 answer
716 views

How should you reshape data before feeding it to LSTM layers?

I was curious if anyone had any advice on how to reshape data for a recurrent neural network. What I've been doing is array.reshape(len(X_train), # of points in time, # of features) And then in the ...
Christopher Palmer's user avatar
-1 votes
1 answer
49 views

Is this the right approach to preprocessing data for artificial neural-networks? [closed]

I recently participated in a competitive "hackathon" with the problem being binary classification of overall satisfaction for travelers. The dataset mostly consisted of survey questions and ...
FIRE KhaN's user avatar
0 votes
2 answers
76 views

How to represent multi-label colours in one-hot encoding?

Say I want to predict the price of a gemstone based on its colour. I have two options: averaging over its colour on an RGB scale, or using its textual description. If I was to choose the latter, how ...
GeorgeWTrump's user avatar