Questions tagged [datasets]

For questions related to sets of data and their use in AI.

96 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5 votes
0 answers
2k views

How many training data is required for GAN?

I'm beginning to study and implement GAN to generate more datasets. I'll just try to experiment with state-of-the-art GAN models as described here https://paperswithcode.com/sota/image-generation-on-...
user avatar
  • 195
5 votes
0 answers
122 views

Are there any easy ways to create annotated training images for object detection?

For the purposes of object detection, are there any easy ways to create annotated training images? For example, if we have $10,000$ images and want to draw bounding boxes on 2 objects for each image, ...
user avatar
  • 61
4 votes
2 answers
45 views

Can I use self-driving car's data set for left-hand drive cars which drive on the right lane for right-hand cars which drive on the left lane?

Can I use self-driving car's data set for left-hand drive cars which drive on the right lane for right-hand self-driving cars which drive on the left lane?
user avatar
  • 41
4 votes
2 answers
87 views

How should I select the features for predicting diseases (in particular when patients specify their health issues)?

My aim is to train a model for predicting diseases. Now, according to this Wikipedia article, diseases are classified based on the following criteria in general: Causes (of the disease) Pathogenesis (...
user avatar
3 votes
2 answers
228 views

How to fix time dimension in time varying data-sets using deep learning model for classification?

Dataset Description I am working on the famous ABIDE Autism Datasets. The dataset is very big in the sense that it has more than 1000 subjects containing half of them as autistic and the other half as ...
user avatar
  • 81
3 votes
0 answers
78 views

How to train CNN such it eliminate dependent features and focuses on independent ones?

How we should train a CNN model when training dataset contains only limited number of cases, and the trained model is supposed to predict class (label) for several other cases, which has not seen ...
user avatar
  • 131
2 votes
0 answers
47 views

Does the Frechet Inception Distance (FID) consider color?

I was wondering if the Frechet inception distance for two colored datasets would be the same than the FID calculated for the same datasets converted to grayscale. I know that it depends on the feature ...
user avatar
  • 211
2 votes
0 answers
19 views

Do any practical deep learning algorithms deal with tensors containing non-real entries?

In deep learning, most of the applications are from text and images. Both text and images can be converted into a tensor of real numbers. Other than both mentioned above, there may be some other real-...
user avatar
  • 3,201
2 votes
0 answers
156 views

In few-shot classification, should I use my custom dataset as the validation dataset and mini-ImageNet as the training dataset?

I am new to few-shot learning, and I wanted to get a hands-on understanding of it, using Reptile algorithm, applied to my custom dataset. My custom dataset has 30 categories, with 5 images per ...
user avatar
2 votes
0 answers
52 views

How can I formulate a prediction problem (given labeled data) as an RL problem and solve it with Q-learning?

One of my friends sent me a problem he was working on lately, and I couldn't help but I wonder how could it be solved using Q-learning. The statement is as follows: Given the following datasets, the ...
user avatar
  • 21
2 votes
0 answers
26 views

How does sampling works in case of imbalanced image datasets?

I am solving a problem of image classification of the image dataset for 3 classes. Dataset is highly imbalanced. How will sampling (either over- or under-sampling) work in that case? Should I remove (...
user avatar
  • 121
2 votes
0 answers
55 views

Why is ANFIS important in general?

I am actually working with the iris dataset from sklearn and try to understand the ANFIS-Package for python. But that does not really matter! I have a more general question. During thinking about ...
user avatar
2 votes
0 answers
30 views

Will structured knowledge bases continue to be used in question answering with the likes of BERT gaining popularity?

This may come across as an open and opinion-based question, I definitely want to hear expert opinions on the subject, but I am also looking for references to materials that I can read deeply. One of ...
user avatar
  • 121
2 votes
0 answers
40 views

Labeling for multilabel image classification

with a friend of mine, we got in an argument over how to label images for multi-label. Note: Groups of a species and the species of catfish is important to recognize. The labels are: 'I': an ...
user avatar
  • 91
2 votes
0 answers
42 views

How can I compare EEG data with accelerometer data in 1 algorithm?

I have frequency EEG data from fall and non-fall events and I am trying to incorporate it with accelerometer data that was collected at the same time. One approach is, of course, to use two separate ...
user avatar
  • 21
2 votes
0 answers
96 views

Suggestion for finding the stable regions in spiral galaxy data?

I am working with a data set that consists of the actual pitch angle (given as PA(Y)) and the pitch angle at each radii (listed from 1 to 217). In the image below, ...
user avatar
2 votes
1 answer
114 views

How to perform binary classification when one class is more predominant than the other?

Assuming we have big $m \times n$ input dataset, with $m \times 1$ output vector. It's a classification problem with only two possible values: either $1$ or $0$. Now, the problem is that almost all ...
user avatar
  • 21
2 votes
0 answers
75 views

How to learn using DDPG in python solely using a timeseries datasets

I have a lengthy timeseries datasets which contains several variables (from sensors etc) to be classified as actions or states. Providing they are successfully done, I want to learn a control policy ...
user avatar
  • 131
2 votes
1 answer
67 views

Is it compulsary to normalize the dataset if doing so can negatively impact a Binary Logistic regression performance?

I am using raw data set with 4 feature variables (Total Cholesterol, Systolic Blood Pressure, Diastolic Blood Pressure, and Cigraeette count) to do a Binominal Classification (find stroke likelihood) ...
user avatar
  • 51
2 votes
0 answers
26 views

Train detector : 300 images with 30 objects or 9000 images with one?

so I have this dataset of images of people sitting in a restaurant. I've annotated about 300 images with an average of 30 instances of "person" per image. Now I'm wondering if I should have annotated ...
user avatar
  • 121
2 votes
0 answers
66 views

How to choose our data set wisely?

I have a couple of questions and I was wondering if you could answer them. I have a bunch of images of the cars, side view only. I want to train the model with those images. My objects of interest ...
user avatar
  • 229
2 votes
0 answers
107 views

Generate QA dataset from large text corpus

I have a corpus of a domain data in form of 10-15 books pdf and some articles and my end-goal is to make a question-answering system particular to that domain. For that, I would need a dataset on Q/A ...
user avatar
2 votes
0 answers
46 views

Difference between retraining on different portions of data and training initially on larger data set

I have a large data set that doesn't fit in memory and would have to use something like Keras's model.fit_generator if I would like to train the model on all of the ...
user avatar
2 votes
0 answers
588 views

Deep Learning for radio signal classification with DeepSig dataset

I want to see if I can make my Software Defined Radio, SDR, to classify unknown radio signals with the help of an artificial neural network. That is, my SDR outputs a sequence of complex numbers (IQ-...
user avatar
  • 21
2 votes
0 answers
35 views

What methods are there to generate artificial training examples based on existing training examples?

I have a small dataset (117 training examples) and many features (4005). Each of the training examples is binary labeled (healthy / diseased). Each feature represents the connectivity between two ...
user avatar
  • 141
2 votes
1 answer
63 views

Shifting training data

I want to create a neural network and train it on some data, however I want to be able to create a new model without retraining it from the start. An example, I have 1000 data points in my training ...
user avatar
  • 31
2 votes
0 answers
151 views

Classification Learning - Normalization of time series and live usage

UPDATE: The tables look messed up so i put them on pastebin for better visibility. https://pastebin.com/gDX28uVF I am using Neural Network with different learning types (for example Standard ...
user avatar
  • 34
2 votes
0 answers
153 views

Preprocessing of training dataset for machine learning

I'm developing a log analyzer to predict and find errors in an equipment. Each logged data contains the following format: ...
user avatar
2 votes
5 answers
184 views

How do I increase the size of an (almost) balanced dataset?

I am trying to add more data points in my (almost) balanced dataset for training my neural network. I have come across techniques such as SMOTE or Random Over Sampling, but they work best for ...
user avatar
1 vote
2 answers
15 views

Master theorem about polynomial classifiers?

Does anyone know if there is a theorem or counterexample establishing whether or not for any given binary classification task in some finite (possibly large) dimensional vector space of attributes, ...
user avatar
1 vote
1 answer
17 views

Datasets input at model.fit produce unexpected results of training loss vs validation loss

Im trying to train a neural network (VAE) using tensorflow and Im getting different results based on the type of input in the model.fit. When I input arrays I get normal difference between the ...
user avatar
1 vote
1 answer
36 views

What makes a 'good' dataset

for the usage of ML technologies, having a appropriate dataset is arguably the first and fundamental step one has to tackle by either aquiring a dataset from external sources or creating their own. ...
user avatar
  • 11
1 vote
0 answers
19 views

What are the types of data in which the order of instances does matter?

In general, the order of instances in the datasets that are used in machine learning is immaterial. But there are exceptions. Timeseries data is one such exception I know. Consider the following two ...
user avatar
  • 3,201
1 vote
0 answers
29 views

Recommended way to spilt image sequence for training/validation/testing

For object detection tasks I have a few minutes of video footage from a surveillance camera, converted to a sequence of images and ground truth bounding boxes for all people walking by. Now what's the ...
user avatar
  • 33
1 vote
0 answers
40 views

Is it true that real world data is highly discontinuous?

A function $f$ is said to be continuous at a point $c$ if it satisfies three properties: Should be defined at the point $c$ Left and right-hand limits at $c$ must be equal i.e., the limit must exist ...
user avatar
  • 3,201
1 vote
0 answers
32 views

Training on the dataset in parts vs training on the whole dataset

What is the difference between these two situations? are they the same ? #1 : train a model 20 epochs on the whole dataset #2 : divide dataset into n-parts then train the model 20 epochs on each part ...
user avatar
1 vote
0 answers
63 views

Batch normalization for multiple datasets?

I am working on a task of generating synthetic data to help the training of my model. This means that the training is performed on synthetic + real data, and tested on real data. I was told that batch ...
user avatar
  • 211
1 vote
0 answers
46 views

Identifying rotating and resizing letters with background noise

I'm trying to complete a captcha, and here is what it looks like: Between captchas the calligraphy of the letters is the same, but the letters may be resized and rotated. And the background noise (...
user avatar
1 vote
0 answers
11 views

Split on dataset with some shared features?

I have a dataset with financial stock data, some of the features are shared, for example daily gold prices, while the stock price for each individual stock is different, the gold price would be the ...
user avatar
1 vote
0 answers
43 views

Pixel values of segmap in multi-class semantic segmentation

I'm preparing a dataset for a multiclass semantic segmentation using U-Net like architecture. To be precise, I've got it ready but a question came to my mind. How does pixel values of a segmentation ...
user avatar
  • 11
1 vote
0 answers
20 views

Theoretical limits on correlation between classification algorithm performances

Are there any known theoretical bounds, or at least heuristic approaches, regarding the relation or correlation between the performances of any two different classification algorithms? For example, ...
user avatar
1 vote
0 answers
45 views

Is training on single game each time appropriate for an agent to learn to play checkers

I was facing a problem I mentioned in a previous question but after a while, I realize that maybe the problem in the dataset not in the learning rate. I build the dataset from white positions only i.e ...
user avatar
1 vote
0 answers
30 views

How to find distance between 2 points when dimensions are all of different nature?

I have a dataset with four features: the x coordinate the y coordinate the velocity magnitude angle Now, I want to measure the distance between two points in the dataset, taking into account the ...
user avatar
1 vote
0 answers
143 views

Which loss function to choose for imbalanced datasets?

For imbalanced datasets (either in the context of computer vision or NLP), from what I learned, it is good to use a weighted log loss. However, in competitions, the people who are in top positions are ...
user avatar
1 vote
0 answers
50 views

Could the neural network automatically calculate and get different one-to-many quantities relative to their parent quantity?

Let's say I have a primary dataset that its secondary dataset is hundreds to match and group like an one-to-many relationship. I'm new in this world of the AI but my problem is that many child groups ...
user avatar
1 vote
0 answers
317 views

Multilabel stratified split for images/object detection

I am working on an object detection model and have thought of looking into stratified splits for the dataset. Now since I am doing object detection I have a variable number of "labels" for ...
user avatar
1 vote
0 answers
28 views

How is the performance of a CNN trained with monochrome images on image recognition tasks degraded?

For CNN image recognition tasks, like object recognition/face recognition/object segmentation/posture recognition, are there experiment results about how much will the performance be degraded with ...
user avatar
  • 201
1 vote
0 answers
33 views

How can raw data from a motion sensor (like an IMU) reduced to the main points of the data

How can I reduce the caputured movement data of a person in a way, that I have filtered the main features of the movement. Or how can I detect pattern/ main features in that data set? I captured some ...
user avatar
  • 435
1 vote
0 answers
17 views

What are the best datasets available for music information retrieval?

I am interested in doing some work in classification problems in music information retrieval. I know that there are some formats of datasets (such as MIDI, Spectrogram, Piano-roll, MusicXML, etc.) for ...
user avatar
  • 111
1 vote
0 answers
43 views

How can I approximate a function that determines the priority of objects?

I am facing the following supervised learning problem: An object is fully characterized by its position in $R^n$. There are $m$ objects. There are fully observable (i.e. their positions are always ...
user avatar
  • 111