Skip to main content

Questions tagged [data-science]

Use for questions related to Data Science aspects of AI. Generally speaking, only basic Data Science questions should be asked on this Stack, ideally involving the fundamental concepts. For more advanced questions related to Data Science, please use the Data Science stack: https://datascience.stackexchange.com/

Filter by
Sorted by
Tagged with
0 votes
1 answer
29 views

Can I scale subsets of my dataset independently to handle different feature ranges?

I am currently making an LSTM model to predict the change in value of a stock for each day. My dataset consists of 4 years of data for 30 different stocks along with some financial metrics such as RSI,...
Daniel goldney's user avatar
0 votes
0 answers
19 views

Is it possible to automate data mining

I'm a Frontend Dev tasked with Data Insights (Help!) Hey everyone! I'm a fresh graduate working as a front-end developer. This is my first real job, and I'm thrown into the deep end a bit! My first ...
Zihan Azad's user avatar
0 votes
1 answer
30 views

How to estimate Time vs Memory trade-off prior to modelling

It is often the case when the time vs memory trade-off is underestimated prior to using ML/DL for solving a particular task. Taking into account the type, size and format of the available data and ...
Deyan's user avatar
  • 1
0 votes
0 answers
22 views

Is it possible to calculate a GRU RNN in its entirety by hand on a small dataset?

I want to see whether my code works and want to do it by hand to compare the results. How exactly does the memory cell work in the first example seen?
J_Bake's user avatar
  • 11
0 votes
0 answers
10 views

Dimensions and file sizes of AI types (img/txt/sound) that work commercially and projections of future AI based on dimensionality and file simplicity?

Is it accurate that current AI breakthroughs are proportional to low dimensional complexity of datasets and small data throughput? Image and text file renders are 100's of kilobytes, and they are ...
bandybabboon's user avatar
0 votes
0 answers
40 views

Optimal decision with continuous, stochastic signals and rewards

I am performing a task, where I have to decide which projects to pursue at a given point in time, where the projects have different horizons of 30 minutes. At a given point in time, forecasts are made ...
Mathman's user avatar
4 votes
3 answers
1k views

What is a pipeline in machine learning?

I have heard the term "pipeline" used in many different contexts. Now I'm trying to bring some clarity to the terminology: What exactly is a "pipeline" in machine learning?
user946822's user avatar
0 votes
1 answer
249 views

Why aren’t intellectual leaders stepping in to clarify that we know little to nothing about how intelligence works or how to build intelligence?

There is no known way of building intelligence/superintelligence today (often and hereafter referred to as AGI), nor is there a known way towards building it in the future. Unlike fusion reactors, for ...
Dudelstrom's user avatar
0 votes
0 answers
21 views

Is synthetic data just a placebo for immature models?

I apologize for the provocative question, but let me elaborate. I am trying to wrap my head around the logic of synthetic data. When you train a model what you are trying to do is to teach the ground ...
Pigna's user avatar
  • 295
0 votes
0 answers
21 views

In $n$ dimensions, why do we need at least $n+1$ sample points to have covariance of sample data non-singular?

In $n$ dimensions, why do we need at least $n+1$ sample points to have covariance of sample data non-singular?
DSPinfinity's user avatar
1 vote
0 answers
16 views

Calculating i_delta in upper confidence bound algorithm

I wanted to apply upper confidence bound algorithm. There is a 2 dimensional dataset in it Each user in the dataset reviewed (((all))) movies with 0 or 1 not some movies only And i want to get the ...
ZIGO's user avatar
  • 11
0 votes
1 answer
46 views

Second Language for Machine Learning [closed]

I want to pursue Machine Learning and Data Science as a future career and wanted to get your advice on a good second language that will be useful in the ML field. Currently, I am learning Python and ...
Rishab's user avatar
  • 101
2 votes
0 answers
85 views

How does one deal with images that are too large to fit in the GPU memory for doing ML image analysis?

How does one deal with images that are too large to fit in the GPU memory for doing ML image analysis? I am interested in detecting small structures on images which are themselves many GB in size. ...
Luca's user avatar
  • 121
2 votes
0 answers
26 views

How to decide which column has more weightage to output

As per Image we can see Column_A value is directly proportional to output, While Change in value of Column_B has no effects in output. So basically I want to know is there any algorithm where I can ...
Marques's user avatar
  • 21
0 votes
0 answers
64 views

How to handle missing data for an ordinal feature variable for a deep learning model?

BACKGROUND: I am developing a deep learning model in which one feature variable (out of many) is the grade of cancer, an ordinal variable. Below is a breakdown of my data by grade: ...
Snehal Patel's user avatar
0 votes
0 answers
14 views

How to determine whether this situation belongs to data leakage or not

Suppose that I use three features (x1, x2, x3) to predict the value of y. After hyperparameters tuning, the r2 score on train/valid/test set is 0.92, 0.54, 0.55 respectively, it's not so good.(It is ...
Zhang NaiChi's user avatar
0 votes
0 answers
105 views

The Small Set Expansion Hypothesis, this problem was solved or is open problem yet?

I found this problem by article called "Ten Lectures and Forty-Two Open Problems in the Mathematics of Data Science" published at 2016. I`m looking for an open problem at Data Science or/and ...
Leonardo Teramatsu's user avatar
-1 votes
2 answers
258 views

What is the relationship between data science, artificial intelligence,machine learning and computer vision?

I am beginner to this field and i am trying to find big picture and i have tried to explore youtube and google images in this regard. According to my understanding ,machine learning is subset of ...
DSP_CS's user avatar
  • 171
1 vote
0 answers
12 views

how to manage the impact of Covid on building a machine learning model

I need your suggestions for using historical data to build a machine learning model for analyzing the market and build an AI model(tree based model/random forest or regression analysis) for setting ...
Mahsa's user avatar
  • 111
2 votes
2 answers
135 views

LSTM exploding? - multiple parallel time series with multiple variables

I have the following situation: Stock Time_Stamps Feature_1 Feature_2 Feature_n Price Stock_1 2019 0.5 1.0 1.0 100 Stock_1 2020 0.7 1.3 0.9 90 Stock_2 2019 0.3 0.9 1.1 110 Stock_2 2020 0.2 0.8 1....
Sphenoidale's user avatar
2 votes
1 answer
170 views

What are the possible ways to handle imbalance in multi-class image datasets?

Image imbalance is one of the major factor in the performance of DL model. Some of the methods that I found to tackle this are oversampling, under-sampling, SMOTE. Over-sampling has cons as it makes ...
Dilip C M Dept of MCA's user avatar
2 votes
1 answer
57 views

How to tackle the human error made in labeling datasets for classification tasks like facial expression recognition?

I am working on the Facial Expression Recognition Task. One of the most challenging tasks that I faced was human error in labeling the datasets (ex: let's say FER2013). Are there anyways to Handle ...
Dilip C M Dept of MCA's user avatar
1 vote
2 answers
948 views

Musical notes interpretation [closed]

Musical notes Musical notes videos Piano Can AI, Machine learning, Data science, Computer vision, image processing technologies assist in interpreting musical notes ? Input dataset : Musical notes ...
Prashant Akerkar's user avatar
0 votes
1 answer
52 views

Generating a dataset from data with "assumed" lables

I've got a task similar to the following: Out of x amount of people, I need to predict, who could be a good athlete and who not. The thing is, I don't have data on the athletic performance of those ...
Chris's user avatar
  • 103
1 vote
1 answer
54 views

How can I address missing values for LSTM?

I'm a student and writing my first paper for submission on conference. I have a question there is a dataset below. this is temporal-spatial dataset. ...
Dae-Young Park's user avatar
2 votes
0 answers
173 views

Taking a machine learning model to production\deployment

I've designed a machine learning model for the predictive maintenance of machines. The data used for training and testing the ML model is the data from various sensors connected to various parts of ...
santobedi's user avatar
2 votes
1 answer
102 views

Is there a clustering algorithm that can make n clusters and the n+1 "others" cluster?

As far as I know all clustering algorithms assume that all delivered data points have to find its cluster. My question is, is there an algorithm that could focus only on n clusters (number stated by ...
GKozinski's user avatar
  • 1,270
1 vote
0 answers
108 views

Does the order of data augmentation and normalization matter?

What is the preferred order of data augmentation and normalization? Is it the former followed by the latter?
shyam vishnu's user avatar
2 votes
0 answers
88 views

Why do I get higher average dice accuracy for less data

I am working on image segmentation of MRI thigh images with deep learning (Unet). I noticed that I get a higher average dice accuracy over my predicted masks if I have less samples in the test data ...
Lis Louise's user avatar
1 vote
0 answers
73 views

Why is the accuracy of my model very low on a separate dataset from the training and test datasets?

I am working on stock price prediction project, I am using the support vector regression (SVR) model for it. As I am splitting my data into train and test, I am getting high accuracy while predicting ...
Debugger's user avatar
2 votes
1 answer
715 views

Is this dataset with only two features suitable for clustering with k-means?

I am working with the K-means clustering algorithm for unsupervised learning. Is the following dataset suitable for the k-means clustering task or not? Why or why not? The dataset has only two ...
Debugger's user avatar
1 vote
0 answers
72 views

Are there any general guidelines for dealing with imbalanced data through upsampling or downsampling?

Are there any general guidelines for dealing with imbalanced data through upsampling/downsampling? This Google developer guide suggests performing downsampling with upweighting, but for the most ...
information_interchange's user avatar
1 vote
0 answers
28 views

What are the best datasets available for music information retrieval?

I am interested in doing some work in classification problems in music information retrieval. I know that there are some formats of datasets (such as MIDI, Spectrogram, Piano-roll, MusicXML, etc.) for ...
Babaji's user avatar
  • 111
2 votes
0 answers
36 views

Possible approaches to dealing with unbalanced dataset and highly biased deep learning algorithm

I have an extremely unbalanced video dataset for a two class video classification problem.All my videos in my current video dataset is $40$ second long with $900p$ resolution.However the dataset is ...
user2359877's user avatar
2 votes
1 answer
72 views

Predicting a day's data

I have a dataset containing timestamp and temperature. For each day, I have 1440 values viz., I have data for every minute of that day(60minutes * 24hrs = 1440). The Dataset looks like this: As an ...
Koushik Raghav S's user avatar
3 votes
0 answers
137 views

Speaker Identification / Recognition for less size audio files

I am working on speaker identification problem using GMM (Gaussian Mixture Model). I have to just identify one user present in the given audio, so for second class noise or silent audio may use or not ...
Posi2's user avatar
  • 368
2 votes
1 answer
235 views

How to define the "Pre-Processing" in machine learning?

Is every process (such as data acquisition, splitting the data for validation, data cleaning, or feature engineering) that is done on the data before we train the model always called the pre-...
malioboro's user avatar
  • 2,819
3 votes
1 answer
889 views

Can alpha-beta pruning be used for applications apart from games?

Can alpha-beta pruning/ minimax be used for systems apart from games? Like for selecting the right customer for a product, etc. (the typical data science problems)? I have seen people do it, but can't ...
kritika's user avatar
  • 33
0 votes
1 answer
56 views

How to calculate the false positives and negatives?

I have a huge amount of data and I want to calculate my false positive and false negative. Is there a software that can help me determine it?
user29622's user avatar
5 votes
3 answers
76 views

What should we do when we have equal observations with different labels?

Suppose we have a labeled data set with columns $A$, $B$, and $C$ and a binary outcome variable $X$. Suppose we have rows as follows: ...
guest_guy's user avatar
1 vote
1 answer
2k views

How to rescale data to its original range after MinMaxScaler? [closed]

I'm using sklearn's MinMaxScaler in order to scale my data down. However, it would be nice to be able to rescale it back to its original range. Is there any way I ...
cjm333's user avatar
  • 63
1 vote
0 answers
32 views

Scikit-Learn: monotoneous quantile estimation

I would like to implement various AI-estimators for quantile estimation for a regression problem. It would be necessary to have non-crossing quantiles, that is larger quantiles would correspond to ...
user823's user avatar
  • 111
2 votes
1 answer
345 views

How do I know if my dataset is ready for a machine learning model?

I am new in this area of Machine Learning and Neural Networks. Currently, I'm taking some courses on Udemy and reading a book about it, but I still have one big question regarding data pre-processing. ...
Pedro de Sá's user avatar
3 votes
1 answer
97 views

How to make a distinction between item feature and environment feature?

My data is stock data with features such as stocks' closing prices.I am curious to know if I can put the economy feature such as 'national interest rate' or 'unemployment rate' besides each stocks' ...
Eiffelbear's user avatar
1 vote
0 answers
18 views

Are hadoop ecosystem tools main goal is to break up large data sets into fast readable files?

I am new to big data theory, and during the past 3 days, I took an official big data course with some of the best instructors available in my country in this domain. The things was little bit obscure ...
alim1990's user avatar
  • 131
1 vote
1 answer
48 views

What data formats/pipelining are best to store and wrangle data which contains both text and float vectors?

Often in NLP project the data points contain both text and float embeddings, and it's very tricky to deal with. CSVs take up a ton of memory and are slow to load. But most the other data formats seem ...
SantoshGupta7's user avatar
0 votes
1 answer
57 views

What types of machine learning model would fit? [closed]

I am working on a supervised machine learning problem where I have more than 10 probably 50 or 100 predicting label categories. Which type of model can be used to work on this type of problem in ...
Par's user avatar
  • 9
0 votes
1 answer
34 views

Loss/accuracy on Synthetic data [closed]

I am trying to understand if there is any difference in the the interpretation of accuracy and loss on synthetic data vs real data.
Vasudevan Visvanathan's user avatar
0 votes
1 answer
272 views

Parameters to calculate affluence in localities of Metro city

I have to calculate the affluence in localities of Metro city. To calculate affluence, I am considering a parameter per capita income. Where I can get a dataset of it? What are other parameters I ...
Posi2's user avatar
  • 368
0 votes
1 answer
162 views

Which field to study to learn & create a.i generated simulations?

I wasn't sure how to title this question so pardon me please. You may have seen at least one video of those "INSANE A.I created simulation of {X} doing {Y & Z} like the following ones: A.I ...
user avatar