Questions tagged [datasets]
For questions related to sets of data and their use in AI.
249
questions
0
votes
0
answers
3
views
What are common benchmarks (simulators and datasets) for testing compositionally in the visual domain?
Learning an object decomposition from a visual scene is a difficult problem for language models describing the scene, for radiance fields that reconstruct the scene, and also for generative models ...
- 156
5
votes
1
answer
258
views
How can I encode angle data to train neural networks?
I am training a neural network where the target data is a vector of angles in radians (between 0 and 2*pi).
I am looking for study material on how to encode this data.
Can you supply me with a book or ...
- 241
1
vote
1
answer
37
views
Can independent datasets be artificially combined for multimodal learning (semi-synthetic data generation)?
BACKGROUND: To apply multimodal machine learning (ML), the various data modalities typically come from the same example (e.g., chest X-ray (modality 1) and ...
- 657
2
votes
1
answer
42
views
Are the "artifacts" in select Keras MNIST training images really there or is my download corrupt?
I'm having fun with a ludicrously well known and used dataset: mnist.
I am doing it with a huge and well known tool: keras.
Please excuse the red dots, something else I was doing. I have otherwise ...
- 283
0
votes
1
answer
30
views
Why does data augmentation using synthetic data generated by one model improve the performance of another model?
I understand from articles like this one that synthetic data generated by one model based on real data can improve the performance of a second model. Can anyone help me understand the intuition behind ...
0
votes
0
answers
44
views
Machine Learning Methods commonly used when data are scarse
It is well-known that deep neural networks require lots of data to perform reliably and well. A commonly-cited statistic is that you need at least 10,000 examples per class for a classification ...
-1
votes
0
answers
12
views
Generating synthetic time series data with limited data
I would like some opinions on my current situation.
I have a set of time series data that I want to forecast. The data however is not very long (around 500 rows) so I was looking into generating many ...
0
votes
1
answer
46
views
To train a mushroom vs. non-mushroom image classifier, which images should comprise the negative class?
I want to train a model that will identify if the image is mushroom and non-mushroom.
If the image is mushroom, I will use another existing model to categorize if its poisonous or edible.
I want to ...
1
vote
3
answers
114
views
Why does MNIST provide only a training and a test set and not a validation set as well?
I was taught that, usually, a dataset has to be divided into three parts:
Training set - for learning purposes
Validation set - for picking the model which minimize the loss on this set
Test test - ...
- 37
2
votes
1
answer
129
views
Is there an image classification dataset where the class depends on spatial relations?
My question is pretty much the one asked above. To clarify a bit further: I have only found datasets that do object localization and that also have relations between the objects annotated (like: "...
- 21
1
vote
2
answers
51
views
Does more data increase training accuracy in neural networks?
I was wondering what is the performance benefit of feeding more data to a machine learning model like a neural network? Like I know one of the benefits is that it increases generalization - testing ...
- 11
1
vote
1
answer
61
views
Feature Engineering on transactional dataset clustering
I have a data set with transactions details from different business (roughly 1 thousand business entities). Each row is a transaction. The structure of the dataset is as follows:
client_id
Sex
Age
...
1
vote
2
answers
28
views
Master theorem about polynomial classifiers?
Does anyone know if there is a theorem or counterexample establishing whether or not for any given binary classification task in some finite (possibly large) dimensional vector space of attributes, ...
0
votes
0
answers
13
views
How to identify pulse pattern in data
So I am using a pulse sensor which is giving out a certain data. When I place my finger on the sensor and see the data plotted in a graph it shows a varying (non-repetitive as no two pulses are ...
- 1
0
votes
1
answer
45
views
Meaning of Large Dataset for machine learning
Some online answers about parameters in machine learning mention that it is dependent on the size of dataset we have (if it is a large dataset or not). Is this size related to the number of samples we ...
0
votes
1
answer
90
views
combine two features in dataset?
I have a data set containing the number of security gaps and the level of that gap for a specific website.
Now suppose I have 2 features in this data set, the first feature is the number of a ...
1
vote
1
answer
58
views
Datasets input at model.fit produce unexpected results of training loss vs validation loss
Im trying to train a neural network (VAE) using tensorflow and Im getting different results based on the type of input in the model.fit.
When I input arrays I get normal difference between the ...
user56546
0
votes
1
answer
30
views
Should I include overlapping (input) Data in my training data
If I have time dependent data and want to predict the relative change for a future time. Should I separate the data so that the input times don't overlap?
With an example:
I have hourly temperature ...
0
votes
1
answer
34
views
What should be taken as random variables in the distributions of datasets?
Consider the following two paragraphs taken from the paper titles Generative Adversarial Nets by Ian J. Goodfellow et.al
#1: Abstract
We propose a new framework for estimating generative models via ...
- 3,371
1
vote
1
answer
58
views
What makes a 'good' dataset
for the usage of ML technologies, having a appropriate dataset is arguably the first and fundamental step one has to tackle by either aquiring a dataset from external sources or creating their own.
...
- 11
0
votes
1
answer
55
views
Is there a way to improve the low-quality data?
I'm on a robotics team and we've been tasked to write a program to differentiate between a live and dead fish. We've been given ~15 minutes of training footage and it's absolutely terrible. It's low ...
- 15
0
votes
0
answers
15
views
Is there any framework for generation of synthetic data where we can set how hard data is to classify?
I am comparing two different training methods of deep neural networks and I was wondering are there any frameworks that can generate x, y datasets but with some input parameter or parameters that can ...
- 1,220
7
votes
2
answers
762
views
Is there an argument against using the (reviewed) predictions of a model as ground truth to further train exactly this model?
I plan to use my predictions as ground truth to continue training my model. These predictions are of course reviewed during this process. Is there an argument against that (reinforcement of slight ...
- 73
0
votes
0
answers
16
views
Which existing model could be used for wind speed and direction prediction?
I am trying to predict the wind speed and wind direction in a graph network for a geographical area. The dataset includes the start and end nodes, the distance between them, and wind speed and ...
- 1
4
votes
1
answer
105
views
What is the total number of actions and rewards count
Reading this two articles about Reinforcement Learning:
Deep Reinforcement Learning with Double Q-learning by Hado van Hasselt et al.
Human-level control through deep reinforcement learning by ...
- 143
0
votes
0
answers
19
views
How to use oxford5k for training?
Generally, we have training data with landmark IDs, their GTs (positive samples), and then separate query images and corresponding positive samples for evaluation.
In the Oxford5k or ROxford5k, one ...
- 101
0
votes
3
answers
109
views
How to deal with an unbalanced dataset?
I'm constructing a feed forward neural network that predicts whether a patient will get a stroke or not. However, my dataset is very unbalanced. Out of 5111 rows, 250 contain patients that have had a ...
- 9
0
votes
1
answer
41
views
How to create a dataset for binary classification
I would like to classify whether a pot of water is boiling or not using a CNN.
Is it enough to take pictures of boiling water using only one pot, or should I use different pots for this to generalize ...
- 123
0
votes
1
answer
46
views
Having the negative cases in the same batch vs. shuffling the dataset
I am working on a model for an NLP task. The model encodes the text and has a regression output layer.
In this task, from each instance (positive), I create several negative cases using a specific ...
- 123
0
votes
1
answer
72
views
Is there a standard term for the following flaw in the data?
I wonder if following characteristic of data has some standard "professional" or scientific term associated with it.
Let's assume that I have a set of dog/cat images labeled 0 for a cat and ...
- 1,220
0
votes
0
answers
28
views
What are the "per image" annotations that are generally used for image datasets in AI?
Computer vision is highly benefited by AI algorithms. Image data is abundantly available. There are different varieties of tasks such as image classification, prediction, segmentation, generation, ...
- 3,371
0
votes
0
answers
35
views
What is all necessary types of data for a bidirectional RNN to learn embeddings?
Bidirectional RNNs are used for generating the semantic vectors of the text at the sentence level and word level.
In order to train a CNN for the classification tasks, images, and labels/outputs are ...
- 3,371
0
votes
0
answers
7
views
Predicting single floats based on set of 2 feature arrays each of 100 values
I am trying to predict audio to video desynchronization based on ser od two arrays of lenght 100 which consist of coresponding audio and video samples.
The problem is that my labels are single floats (...
1
vote
0
answers
20
views
What are the types of data in which the order of instances does matter?
In general, the order of instances in the datasets that are used in machine learning is immaterial. But there are exceptions. Timeseries data is one such exception I know. Consider the following two ...
- 3,371
0
votes
0
answers
11
views
What to predict in a limited transaction dataset?
I have been given a task with a real transaction dataset. The task is to predict something using either logistic regression or simple binary classification.
The columns are as follow:
Transaction ID
...
- 31
2
votes
0
answers
59
views
Does the Frechet Inception Distance (FID) consider color?
I was wondering if the Frechet inception distance
for two colored datasets would be the same than the FID calculated for the same datasets converted to grayscale.
I know that it depends on the feature ...
- 221
1
vote
1
answer
67
views
Is there any simple example for volumetric data except from physics and medicine?
Recently I heard about the term volumetric data. The definition for volumetric data is as follows
#1: Definition
Volumetric data is typically a set S of samples $(x, y, z, v)$,
representing the value ...
- 3,371
2
votes
0
answers
19
views
Do any practical deep learning algorithms deal with tensors containing non-real entries?
In deep learning, most of the applications are from text and images. Both text and images can be converted into a tensor of real numbers.
Other than both mentioned above, there may be some other real-...
- 3,371
0
votes
1
answer
50
views
Generating a dataset from data with "assumed" lables
I've got a task similar to the following:
Out of x amount of people, I need to predict, who could be a good athlete and who not. The thing is, I don't have data on the athletic performance of those ...
- 103
1
vote
1
answer
109
views
Can I use the transformers for the prediction of historical data?
Can I use the transformers for the prediction of wind power with the historical data?
Dataset
Datetime, Ambient temperature (Degree), Dewpoint (Degree), Relative Humidity\n (%), Air Pressure, Wind ...
- 11
3
votes
1
answer
192
views
How to determine the quality of synthetic data?
I'm working on a VAE model to produce synthetic data of X-Ray diffraction spectrums.
I try to figure out how I can measure the quality of the spectrums. The goal would be to produce synthetic data ...
- 33
1
vote
0
answers
29
views
Recommended way to spilt image sequence for training/validation/testing
For object detection tasks I have a few minutes of video footage from a surveillance camera, converted to a sequence of images and ground truth bounding boxes for all people walking by.
Now what's the ...
- 33
2
votes
1
answer
137
views
How do you handle unbalanced image datasets?
I have an image data set on which I am training a CNN. The data set is slightly unbalanced. So, my solution up till now was to delete some images of the majority class.
But I now realize that there ...
- 45
8
votes
3
answers
1k
views
Is it okay to use publicly available Instagram videos to train an AI?
Since I haven't found any good training data for my university project, I want to use pictures and videos from public Instagram profiles. Am I allowed to do that?
- 435
0
votes
0
answers
75
views
What are the different possible usages of the word "i.i.d" in machine learning?
The acronym "iid" stands for "independent and identically distributed". It is a property of a sequence of random variables. You can read here for more details. This question is ...
- 3,371
1
vote
1
answer
103
views
Which of the following probability distribution is generating an iid dataset?
Let $X_1, X_2$ be two discrete random variables. Each random variable takes two values: $1, 2$
The probability distribution $p_1$ over $X_1, X_2$ is given by
$$p_1(X_1=1, X_2 = 1) = \dfrac{1}{4}$$
$$...
- 3,371
1
vote
2
answers
191
views
How can we "draw i.i.d" from any probability distribution?
Consider the following paragraph from 2 Learning in High Dimensions in from of the paper titled Geometric Deep Learning Grids, Groups, Graphs, Geodesics, and Gauges
...
- 3,371
0
votes
0
answers
32
views
Modelling of output neuron for mixed features?
A dataset in artificial intelligence, in general, consists of some features (say $n$). Assume that $m$ among them are output features. I want to model this function using a neural network. So, input ...
- 3,371
0
votes
1
answer
61
views
Why disentangling the features of variation in representation?
Consider the following excerpt from abstract of the research paper titled Better Mixing via Deep Representations by Yoshua Bengio et al.
It has been hypothesized, ...
- 3,371
0
votes
1
answer
126
views
How to handle an unbalanced dataset when training object detection algorithms?
I am training an object detection model, and I have some very highly unbalanced data annotations. I have almost 11,000 images, all with dimensions of 1024 $\times$ 1024.
Within those images I have the ...
- 119