Questions tagged [data-science]

Use for questions related to Data Science aspects of AI. Generally speaking, only basic Data Science questions should be asked on this Stack, ideally involving the fundamental concepts. For more advanced questions related to Data Science, please use the Data Science stack: https://datascience.stackexchange.com/

Filter by
Sorted by
Tagged with
1
vote
1answer
28 views

How can I address missing values for LSTM?

I'm a student and writing my first paper for submission on conference. I have a question there is a dataset below. this is temporal-spatial dataset. ...
2
votes
0answers
79 views

Taking a machine learning model to production\deployment

I've designed a machine learning model for the predictive maintenance of machines. The data used for training and testing the ML model is the data from various sensors connected to various parts of ...
-2
votes
1answer
54 views

Assumptions of a Linear Regression [closed]

I was going through the concept of Linear Regression and ran into the concept of deciding whether a Linear Regression Model is the best fit for your data by 5 assumptions: Linearity Homoscedasticity ...
2
votes
1answer
47 views

Is there a clustering algorithm that can make n clusters and the n+1 "others" cluster?

As far as I know all clustering algorithms assume that all delivered data points have to find its cluster. My question is, is there an algorithm that could focus only on n clusters (number stated by ...
1
vote
0answers
40 views

Does the order of data augmentation and normalization matter?

What is the preferred order of data augmentation and normalization? Is it the former followed by the latter?
0
votes
0answers
22 views

How to improve de-noise algorithm on low signal-to-noise ratio features?

In the following plot, I have features that all have a very small predictive power on y, there is a low signal-to-noise ratio. In order to de-noise them, I tried ...
0
votes
0answers
15 views

Low Signal-To-Noise Ratio Data Processing and Model Choices

Attached is a plot showing that y regresses on individual features: Scatter Plot And as shown, each feature has minuscule predictive power on y. I have 130 features like this, all of them are more or ...
2
votes
0answers
61 views

Why do I get higher average dice accuracy for less data

I am working on image segmentation of MRI thigh images with deep learning (Unet). I noticed that I get a higher average dice accuracy over my predicted masks if I have less samples in the test data ...
1
vote
0answers
39 views

Why is the accuracy of my model very low on a separate dataset from the training and test datasets?

I am working on stock price prediction project, I am using the support vector regression (SVR) model for it. As I am splitting my data into train and test, I am getting high accuracy while predicting ...
2
votes
1answer
144 views

Is this dataset with only two features suitable for clustering with k-means?

I am working with the K-means clustering algorithm for unsupervised learning. Is the following dataset suitable for the k-means clustering task or not? Why or why not? The dataset has only two ...
1
vote
0answers
54 views

Are there any general guidelines for dealing with imbalanced data through upsampling or downsampling?

Are there any general guidelines for dealing with imbalanced data through upsampling/downsampling? This Google developer guide suggests performing downsampling with upweighting, but for the most ...
1
vote
0answers
16 views

What are the best datasets available for music information retrieval?

I am interested in doing some work in classification problems in music information retrieval. I know that there are some formats of datasets (such as MIDI, Spectrogram, Piano-roll, MusicXML, etc.) for ...
2
votes
0answers
33 views

Possible approaches to dealing with unbalanced dataset and highly biased deep learning algorithm

I have an extremely unbalanced video dataset for a two class video classification problem.All my videos in my current video dataset is $40$ second long with $900p$ resolution.However the dataset is ...
2
votes
1answer
37 views

Predicting a day's data

I have a dataset containing timestamp and temperature. For each day, I have 1440 values viz., I have data for every minute of that day(60minutes * 24hrs = 1440). The Dataset looks like this: As an ...
3
votes
0answers
125 views

Speaker Identification / Recognition for less size audio files

I am working on speaker identification problem using GMM (Gaussian Mixture Model). I have to just identify one user present in the given audio, so for second class noise or silent audio may use or not ...
2
votes
1answer
108 views

How to define the "Pre-Processing" in machine learning?

Is every process (such as data acquisition, splitting the data for validation, data cleaning, or feature engineering) that is done on the data before we train the model always called the pre-...
3
votes
1answer
248 views

Can alpha-beta pruning be used for applications apart from games?

Can alpha-beta pruning/ minimax be used for systems apart from games? Like for selecting the right customer for a product, etc. (the typical data science problems)? I have seen people do it, but can't ...
0
votes
1answer
44 views

How to calculate the false positives and negatives?

I have a huge amount of data and I want to calculate my false positive and false negative. Is there a software that can help me determine it?
5
votes
3answers
60 views

What should we do when we have equal observations with different labels?

Suppose we have a labeled data set with columns $A$, $B$, and $C$ and a binary outcome variable $X$. Suppose we have rows as follows: ...
1
vote
1answer
154 views

How to rescale data to its original range after MinMaxScaler? [closed]

I'm using sklearn's MinMaxScaler in order to scale my data down. However, it would be nice to be able to rescale it back to its original range. Is there any way I ...
1
vote
0answers
20 views

Scikit-Learn: monotoneous quantile estimation

I would like to implement various AI-estimators for quantile estimation for a regression problem. It would be necessary to have non-crossing quantiles, that is larger quantiles would correspond to ...
2
votes
1answer
58 views

How do I know if my dataset is ready for a machine learning model?

I am new in this area of Machine Learning and Neural Networks. Currently, I'm taking some courses on Udemy and reading a book about it, but I still have one big question regarding data pre-processing. ...
3
votes
1answer
66 views

How to make a distinction between item feature and environment feature?

My data is stock data with features such as stocks' closing prices.I am curious to know if I can put the economy feature such as 'national interest rate' or 'unemployment rate' besides each stocks' ...
1
vote
0answers
12 views

Are hadoop ecosystem tools main goal is to break up large data sets into fast readable files?

I am new to big data theory, and during the past 3 days, I took an official big data course with some of the best instructors available in my country in this domain. The things was little bit obscure ...
1
vote
1answer
44 views

What data formats/pipelining are best to store and wrangle data which contains both text and float vectors?

Often in NLP project the data points contain both text and float embeddings, and it's very tricky to deal with. CSVs take up a ton of memory and are slow to load. But most the other data formats seem ...
1
vote
1answer
43 views

What types of machine learning model would fit?

I am working on a supervised machine learning problem where I have more than 10 probably 50 or 100 predicting label categories. Which type of model can be used to work on this type of problem in ...
1
vote
1answer
29 views

Loss/accuracy on Synthetic data

I am trying to understand if there is any difference in the the interpretation of accuracy and loss on synthetic data vs real data.
0
votes
1answer
229 views

Parameters to calculate affluence in localities of Metro city

I have to calculate the affluence in localities of Metro city. To calculate affluence, I am considering a parameter per capita income. Where I can get a dataset of it? What are other parameters I ...
0
votes
1answer
116 views

Which field to study to learn & create a.i generated simulations?

I wasn't sure how to title this question so pardon me please. You may have seen at least one video of those "INSANE A.I created simulation of {X} doing {Y & Z} like the following ones: A.I ...
2
votes
0answers
72 views

An idea I had to create the first Humanoid using Deep Learning

I've come up with an idea on how we could use a combination of Deep Learning and body sensors to create a walking talking living humanoid. Here goes: First, we will recruit 1 billion people and have ...
1
vote
1answer
65 views

Help with Novelty Recognition and Binary Classification for Emotion Recognition

I’m looking for advice regarding my ML project. Using a special wristband, I am able to collect a bunch of physiological data from human subjects. I want to develop an application to recognize when ...
1
vote
2answers
71 views

What is the impact of scaling the features on the performance of the model?

I am trying to generate a model that uses several physicochemical properties of a molecule (including number of atoms, number of rings, volume, etc.) to predict a numeric value $Y$. I would like to ...
0
votes
1answer
40 views

is it possible to train several Neural Networks on different types of data and combine them?

I want to create a NHL game predictor and have already trained one neural network on game data. What I would like to do is train another model on player seasonal/game data and combine the two models ...
1
vote
2answers
27 views

How can I evaluate the performance of a system that generates text?

I am preparing to perform research comparing the performance of two different systems that probabilistically generate the next word of an input sentence. For example, given the word 'the', a system ...
2
votes
2answers
99 views

Ensemble models - XGboost

I am building 2 models using XGboost, one with x number of parameters and the other with y number of parameters of the data set. It is a classification problem. A yes-yes, no-no case is easy, but ...
5
votes
3answers
132 views

Is there a Python API for manipulating publicly available datasets? [closed]

In my daily machine learning / deep learning workflow, I often want to interact with a dataset in my code. Specifically, I would like to be able to load some module/package which can make sure that I ...
1
vote
2answers
138 views

Is it necessary to know the details behind the AI algorithms and models?

I am interested in the field of artificial intelligence. I began by learning the various machine learning algorithms. The maths behind some were quite hard. For example, back-propagation in ...
1
vote
1answer
81 views

Automatic prediction of whether a customer will come into the shop or not

So as my university project I am planning to make a prediction system as described in the title. My current idea is to use the age/gender classifier and run it on a video(taken in front of a shop) ...
2
votes
0answers
314 views

How do to mitigate or design out hidden feedback loops when designing ML systems?

Two months ago, I've found myself working on a churn detection problem which can be briefly described as follows: Assume the current date is N Use customer behavior for N-1,..N-x dates to develop ...
0
votes
1answer
99 views

multi vs one prediction using Regression

I was trying to build a prediction system where I have the input data arranged in multiple columns. The input data would be of the type where I have weather, service type (bronze, silver, gold), size ...
2
votes
0answers
279 views

Deep NN architecture for predicting a matrix from two matrices

Recently my friend asked me a question: having two input matrices X and Y (each size NxD) where D >> N, and ground truth matrix Z of size DxD, what deep architecture shall I use to learn a deep model ...
6
votes
1answer
4k views

How to implement exploration function and learning rate in Q Learning

I'm trying to implement Q-learning (state-based representation and no neural / deep stuff) but I'm having a hard time getting it to learn anything. I believe my issue is with the exploration function ...
1
vote
1answer
86 views

What are the most challenging tasks aiming to achieve the lowest error rate?

For example there is the MNIST database which is used to test artificial neural network (ANN), however it's not so challenging, because some hierarchical systems of convolutional neural networks ...