Questions tagged [data-science]

Use for questions related to Data Science aspects of AI. Generally speaking, only basic Data Science questions should be asked on this Stack, ideally involving the fundamental concepts. For more advanced questions related to Data Science, please use the Data Science stack: https://datascience.stackexchange.com/

Filter by
Sorted by
Tagged with
0 votes
0 answers
23 views

Neural Network learning XOR. I collected Data on my networks convergence. Is this expected behavior?

I build a neural network from scratch to get a better understanding of the fundamentals of machine learning. The network contains a bias for each neuron and calculates the final error via the mean ...
user avatar
  • 101
0 votes
0 answers
12 views

Deep learning to fill sequence elements missing at random

I have the following problem setup: There is a list of floats (between -1 and 1) that is about 768*2 in length. The values of the floats are features that depend on two documents, the first 768 ...
user avatar
1 vote
2 answers
313 views

Musical notes interpretation [closed]

Musical notes Musical notes videos Piano Can AI, Machine learning, Data science, Computer vision, image processing technologies assist in interpreting musical notes ? Input dataset : Musical notes ...
user avatar
0 votes
0 answers
19 views

Training the Model properly

I am relatively new to Machine Learning. I want to build a good stock price prediction neural network model and I am using features that depend on the price as well as features that do not depend on ...
user avatar
0 votes
1 answer
47 views

Generating a dataset from data with "assumed" lables

I've got a task similar to the following: Out of x amount of people, I need to predict, who could be a good athlete and who not. The thing is, I don't have data on the athletic performance of those ...
user avatar
  • 103
1 vote
1 answer
33 views

How can I address missing values for LSTM?

I'm a student and writing my first paper for submission on conference. I have a question there is a dataset below. this is temporal-spatial dataset. ...
user avatar
2 votes
0 answers
87 views

Taking a machine learning model to production\deployment

I've designed a machine learning model for the predictive maintenance of machines. The data used for training and testing the ML model is the data from various sensors connected to various parts of ...
user avatar
2 votes
1 answer
54 views

Is there a clustering algorithm that can make n clusters and the n+1 "others" cluster?

As far as I know all clustering algorithms assume that all delivered data points have to find its cluster. My question is, is there an algorithm that could focus only on n clusters (number stated by ...
user avatar
  • 1,029
1 vote
0 answers
62 views

Does the order of data augmentation and normalization matter?

What is the preferred order of data augmentation and normalization? Is it the former followed by the latter?
user avatar
2 votes
0 answers
65 views

Why do I get higher average dice accuracy for less data

I am working on image segmentation of MRI thigh images with deep learning (Unet). I noticed that I get a higher average dice accuracy over my predicted masks if I have less samples in the test data ...
user avatar
1 vote
0 answers
41 views

Why is the accuracy of my model very low on a separate dataset from the training and test datasets?

I am working on stock price prediction project, I am using the support vector regression (SVR) model for it. As I am splitting my data into train and test, I am getting high accuracy while predicting ...
user avatar
2 votes
1 answer
267 views

Is this dataset with only two features suitable for clustering with k-means?

I am working with the K-means clustering algorithm for unsupervised learning. Is the following dataset suitable for the k-means clustering task or not? Why or why not? The dataset has only two ...
user avatar
1 vote
0 answers
59 views

Are there any general guidelines for dealing with imbalanced data through upsampling or downsampling?

Are there any general guidelines for dealing with imbalanced data through upsampling/downsampling? This Google developer guide suggests performing downsampling with upweighting, but for the most ...
user avatar
1 vote
0 answers
16 views

What are the best datasets available for music information retrieval?

I am interested in doing some work in classification problems in music information retrieval. I know that there are some formats of datasets (such as MIDI, Spectrogram, Piano-roll, MusicXML, etc.) for ...
user avatar
  • 111
2 votes
0 answers
34 views

Possible approaches to dealing with unbalanced dataset and highly biased deep learning algorithm

I have an extremely unbalanced video dataset for a two class video classification problem.All my videos in my current video dataset is $40$ second long with $900p$ resolution.However the dataset is ...
user avatar
2 votes
1 answer
47 views

Predicting a day's data

I have a dataset containing timestamp and temperature. For each day, I have 1440 values viz., I have data for every minute of that day(60minutes * 24hrs = 1440). The Dataset looks like this: As an ...
user avatar
3 votes
0 answers
128 views

Speaker Identification / Recognition for less size audio files

I am working on speaker identification problem using GMM (Gaussian Mixture Model). I have to just identify one user present in the given audio, so for second class noise or silent audio may use or not ...
user avatar
  • 328
2 votes
1 answer
133 views

How to define the "Pre-Processing" in machine learning?

Is every process (such as data acquisition, splitting the data for validation, data cleaning, or feature engineering) that is done on the data before we train the model always called the pre-...
user avatar
  • 2,511
3 votes
1 answer
357 views

Can alpha-beta pruning be used for applications apart from games?

Can alpha-beta pruning/ minimax be used for systems apart from games? Like for selecting the right customer for a product, etc. (the typical data science problems)? I have seen people do it, but can't ...
user avatar
  • 33
0 votes
1 answer
45 views

How to calculate the false positives and negatives?

I have a huge amount of data and I want to calculate my false positive and false negative. Is there a software that can help me determine it?
user avatar
5 votes
3 answers
63 views

What should we do when we have equal observations with different labels?

Suppose we have a labeled data set with columns $A$, $B$, and $C$ and a binary outcome variable $X$. Suppose we have rows as follows: ...
user avatar
1 vote
1 answer
319 views

How to rescale data to its original range after MinMaxScaler? [closed]

I'm using sklearn's MinMaxScaler in order to scale my data down. However, it would be nice to be able to rescale it back to its original range. Is there any way I ...
user avatar
  • 63
1 vote
0 answers
23 views

Scikit-Learn: monotoneous quantile estimation

I would like to implement various AI-estimators for quantile estimation for a regression problem. It would be necessary to have non-crossing quantiles, that is larger quantiles would correspond to ...
user avatar
  • 11
2 votes
1 answer
72 views

How do I know if my dataset is ready for a machine learning model?

I am new in this area of Machine Learning and Neural Networks. Currently, I'm taking some courses on Udemy and reading a book about it, but I still have one big question regarding data pre-processing. ...
user avatar
3 votes
1 answer
72 views

How to make a distinction between item feature and environment feature?

My data is stock data with features such as stocks' closing prices.I am curious to know if I can put the economy feature such as 'national interest rate' or 'unemployment rate' besides each stocks' ...
user avatar
1 vote
0 answers
13 views

Are hadoop ecosystem tools main goal is to break up large data sets into fast readable files?

I am new to big data theory, and during the past 3 days, I took an official big data course with some of the best instructors available in my country in this domain. The things was little bit obscure ...
user avatar
  • 131
1 vote
1 answer
47 views

What data formats/pipelining are best to store and wrangle data which contains both text and float vectors?

Often in NLP project the data points contain both text and float embeddings, and it's very tricky to deal with. CSVs take up a ton of memory and are slow to load. But most the other data formats seem ...
user avatar
0 votes
1 answer
46 views

What types of machine learning model would fit? [closed]

I am working on a supervised machine learning problem where I have more than 10 probably 50 or 100 predicting label categories. Which type of model can be used to work on this type of problem in ...
user avatar
  • 9
0 votes
1 answer
30 views

Loss/accuracy on Synthetic data [closed]

I am trying to understand if there is any difference in the the interpretation of accuracy and loss on synthetic data vs real data.
user avatar
0 votes
1 answer
238 views

Parameters to calculate affluence in localities of Metro city

I have to calculate the affluence in localities of Metro city. To calculate affluence, I am considering a parameter per capita income. Where I can get a dataset of it? What are other parameters I ...
user avatar
  • 328
0 votes
1 answer
133 views

Which field to study to learn & create a.i generated simulations?

I wasn't sure how to title this question so pardon me please. You may have seen at least one video of those "INSANE A.I created simulation of {X} doing {Y & Z} like the following ones: A.I ...
user avatar
1 vote
1 answer
75 views

Help with Novelty Recognition and Binary Classification for Emotion Recognition

I’m looking for advice regarding my ML project. Using a special wristband, I am able to collect a bunch of physiological data from human subjects. I want to develop an application to recognize when ...
user avatar
2 votes
1 answer
92 views

What is the impact of scaling the features on the performance of the model?

I am trying to generate a model that uses several physicochemical properties of a molecule (including number of atoms, number of rings, volume, etc.) to predict a numeric value $Y$. I would like to ...
user avatar
  • 21
0 votes
1 answer
50 views

is it possible to train several Neural Networks on different types of data and combine them?

I want to create a NHL game predictor and have already trained one neural network on game data. What I would like to do is train another model on player seasonal/game data and combine the two models ...
user avatar
1 vote
2 answers
33 views

How can I evaluate the performance of a system that generates text?

I am preparing to perform research comparing the performance of two different systems that probabilistically generate the next word of an input sentence. For example, given the word 'the', a system ...
user avatar
2 votes
2 answers
107 views

Ensemble models - XGboost

I am building 2 models using XGboost, one with x number of parameters and the other with y number of parameters of the data set. It is a classification problem. A yes-yes, no-no case is easy, but ...
user avatar
1 vote
2 answers
149 views

Is it necessary to know the details behind the AI algorithms and models?

I am interested in the field of artificial intelligence. I began by learning the various machine learning algorithms. The maths behind some were quite hard. For example, back-propagation in ...
user avatar
  • 181
1 vote
1 answer
82 views

Automatic prediction of whether a customer will come into the shop or not

So as my university project I am planning to make a prediction system as described in the title. My current idea is to use the age/gender classifier and run it on a video(taken in front of a shop) ...
user avatar
2 votes
0 answers
356 views

How do to mitigate or design out hidden feedback loops when designing ML systems?

Two months ago, I've found myself working on a churn detection problem which can be briefly described as follows: Assume the current date is N Use customer behavior for N-1,..N-x dates to develop ...
user avatar
  • 63
0 votes
1 answer
114 views

multi vs one prediction using Regression

I was trying to build a prediction system where I have the input data arranged in multiple columns. The input data would be of the type where I have weather, service type (bronze, silver, gold), size ...
user avatar
2 votes
0 answers
315 views

Deep NN architecture for predicting a matrix from two matrices

Recently my friend asked me a question: having two input matrices X and Y (each size NxD) where D >> N, and ground truth matrix Z of size DxD, what deep architecture shall I use to learn a deep model ...
user avatar
7 votes
1 answer
4k views

How to implement exploration function and learning rate in Q Learning

I'm trying to implement Q-learning (state-based representation and no neural / deep stuff) but I'm having a hard time getting it to learn anything. I believe my issue is with the exploration function ...
user avatar
1 vote
1 answer
93 views

What are the most challenging tasks aiming to achieve the lowest error rate?

For example there is the MNIST database which is used to test artificial neural network (ANN), however it's not so challenging, because some hierarchical systems of convolutional neural networks ...
user avatar
  • 10k