Questions tagged [datasets]
For questions related to sets of data and their use in AI.
236
questions
0
votes
0
answers
15
views
Using logical rules in autonomous driving dataset(Argoverse, nuscenes etc) for trajectory prediction [closed]
I have seen some papers which convert traffic rules into logical ones (first order logic or temporal) as a kind of representation of rules knowledge. Like in the paper 'Formalization of Interstate ...
- 26
0
votes
1
answer
47
views
Is there a way to improve the low-quality data?
I'm on a robotics team and we've been tasked to write a program to differentiate between a live and dead fish. We've been given ~15 minutes of training footage and it's absolutely terrible. It's low ...
- 1
0
votes
0
answers
14
views
Is there any framework for generation of synthetic data where we can set how hard data is to classify?
I am comparing two different training methods of deep neural networks and I was wondering are there any frameworks that can generate x, y datasets but with some input parameter or parameters that can ...
- 1,039
7
votes
2
answers
746
views
Is there an argument against using the (reviewed) predictions of a model as ground truth to further train exactly this model?
I plan to use my predictions as ground truth to continue training my model. These predictions are of course reviewed during this process. Is there an argument against that (reinforcement of slight ...
- 73
0
votes
0
answers
15
views
Which existing model could be used for wind speed and direction prediction?
I am trying to predict the wind speed and wind direction in a graph network for a geographical area. The dataset includes the start and end nodes, the distance between them, and wind speed and ...
- 1
3
votes
1
answer
47
views
What is the total number of actions and rewards count
Reading this two articles about Reinforcement Learning:
Deep Reinforcement Learning with Double Q-learning by Hado van Hasselt et al.
Human-level control through deep reinforcement learning by ...
- 133
0
votes
0
answers
16
views
How to use oxford5k for training?
Generally, we have training data with landmark IDs, their GTs (positive samples), and then separate query images and corresponding positive samples for evaluation.
In the Oxford5k or ROxford5k, one ...
- 101
0
votes
3
answers
81
views
How to deal with an unbalanced dataset?
I'm constructing a feed forward neural network that predicts whether a patient will get a stroke or not. However, my dataset is very unbalanced. Out of 5111 rows, 250 contain patients that have had a ...
- 9
0
votes
1
answer
28
views
How to create a dataset for binary classification
I would like to classify whether a pot of water is boiling or not using a CNN.
Is it enough to take pictures of boiling water using only one pot, or should I use different pots for this to generalize ...
- 23
0
votes
1
answer
30
views
Having the negative cases in the same batch vs. shuffling the dataset
I am working on a model for an NLP task. The model encodes the text and has a regression output layer.
In this task, from each instance (positive), I create several negative cases using a specific ...
- 123
0
votes
1
answer
70
views
Is there a standard term for the following flaw in the data?
I wonder if following characteristic of data has some standard "professional" or scientific term associated with it.
Let's assume that I have a set of dog/cat images labeled 0 for a cat and ...
- 1,039
0
votes
0
answers
28
views
What are the "per image" annotations that are generally used for image datasets in AI?
Computer vision is highly benefited by AI algorithms. Image data is abundantly available. There are different varieties of tasks such as image classification, prediction, segmentation, generation, ...
- 3,099
0
votes
0
answers
33
views
What is all necessary types of data for a bidirectional RNN to learn embeddings?
Bidirectional RNNs are used for generating the semantic vectors of the text at the sentence level and word level.
In order to train a CNN for the classification tasks, images, and labels/outputs are ...
- 3,099
0
votes
0
answers
7
views
Predicting single floats based on set of 2 feature arrays each of 100 values
I am trying to predict audio to video desynchronization based on ser od two arrays of lenght 100 which consist of coresponding audio and video samples.
The problem is that my labels are single floats (...
1
vote
0
answers
19
views
What are the types of data in which the order of instances does matter?
In general, the order of instances in the datasets that are used in machine learning is immaterial. But there are exceptions. Timeseries data is one such exception I know. Consider the following two ...
- 3,099
0
votes
0
answers
11
views
What to predict in a limited transaction dataset?
I have been given a task with a real transaction dataset. The task is to predict something using either logistic regression or simple binary classification.
The columns are as follow:
Transaction ID
...
- 11
1
vote
0
answers
40
views
Does the Frechet Inception Distance (FID) consider color?
I was wondering if the Frechet inception distance
for two colored datasets would be the same than the FID calculated for the same datasets converted to grayscale.
I know that it depends on the feature ...
- 201
1
vote
1
answer
53
views
Is there any simple example for volumetric data except from physics and medicine?
Recently I heard about the term volumetric data. The definition for volumetric data is as follows
#1: Definition
Volumetric data is typically a set S of samples $(x, y, z, v)$,
representing the value ...
- 3,099
2
votes
0
answers
19
views
Do any practical deep learning algorithms deal with tensors containing non-real entries?
In deep learning, most of the applications are from text and images. Both text and images can be converted into a tensor of real numbers.
Other than both mentioned above, there may be some other real-...
- 3,099
0
votes
1
answer
47
views
Generating a dataset from data with "assumed" lables
I've got a task similar to the following:
Out of x amount of people, I need to predict, who could be a good athlete and who not. The thing is, I don't have data on the athletic performance of those ...
- 103
1
vote
1
answer
57
views
Can I use the transformers for the prediction of historical data?
Can I use the transformers for the prediction of wind power with the historical data?
Dataset
Datetime, Ambient temperature (Degree), Dewpoint (Degree), Relative Humidity\n (%), Air Pressure, Wind ...
- 11
0
votes
0
answers
18
views
What are the pros and cons of using a normal positional encoding in an adjacency matrix?
I understand that a normal positional encoding helps a transformer to understand pictures better and that it allows the (otherwise permutational invariant transformer-network) to create relationships ...
3
votes
1
answer
127
views
How to determine the quality of synthetic data?
I'm working on a VAE model to produce synthetic data of X-Ray diffraction spectrums.
I try to figure out how I can measure the quality of the spectrums. The goal would be to produce synthetic data ...
- 33
1
vote
0
answers
27
views
Recommended way to spilt image sequence for training/validation/testing
For object detection tasks I have a few minutes of video footage from a surveillance camera, converted to a sequence of images and ground truth bounding boxes for all people walking by.
Now what's the ...
- 33
2
votes
1
answer
81
views
How do you handle unbalanced image datasets?
I have an image data set on which I am training a CNN. The data set is slightly unbalanced. So, my solution up till now was to delete some images of the majority class.
But I now realize that there ...
- 45
8
votes
3
answers
844
views
Is it okay to use publicly available Instagram videos to train an AI?
Since I haven't found any good training data for my university project, I want to use pictures and videos from public Instagram profiles. Am I allowed to do that?
- 405
0
votes
0
answers
66
views
What are the different possible usages of the word "i.i.d" in machine learning?
The acronym "iid" stands for "independent and identically distributed". It is a property of a sequence of random variables. You can read here for more details. This question is ...
- 3,099
1
vote
1
answer
93
views
Which of the following probability distribution is generating an iid dataset?
Let $X_1, X_2$ be two discrete random variables. Each random variable takes two values: $1, 2$
The probability distribution $p_1$ over $X_1, X_2$ is given by
$$p_1(X_1=1, X_2 = 1) = \dfrac{1}{4}$$
$$...
- 3,099
1
vote
2
answers
118
views
How can we "draw i.i.d" from any probability distribution?
Consider the following paragraph from 2 Learning in High Dimensions in from of the paper titled Geometric Deep Learning Grids, Groups, Graphs, Geodesics, and Gauges
...
- 3,099
0
votes
0
answers
31
views
Modelling of output neuron for mixed features?
A dataset in artificial intelligence, in general, consists of some features (say $n$). Assume that $m$ among them are output features. I want to model this function using a neural network. So, input ...
- 3,099
0
votes
1
answer
51
views
Why disentangling the features of variation in representation?
Consider the following excerpt from abstract of the research paper titled Better Mixing via Deep Representations by Yoshua Bengio et al.
It has been hypothesized, ...
- 3,099
0
votes
0
answers
39
views
What are mathematically the factors of variation in deep learning?
The following paragraph from an answer tells us about factors of variation
Factors of variation are some factors which determine varieties in observed data. If that factors change, the behaviour of ...
- 3,099
0
votes
1
answer
75
views
How to handle an unbalanced dataset when training object detection algorithms?
I am training an object detection model, and I have some very highly unbalanced data annotations. I have almost 11,000 images, all with dimensions of 1024 $\times$ 1024.
Within those images I have the ...
- 119
0
votes
0
answers
47
views
What's the best way to feed stories to a neural network?
I'm trying to train a model that would generate stories. I have a dataset of 2000 stories prepared. They are tokenized and one-hot encoded. I can't load them all at once as a one big dataset, because ...
- 1
0
votes
2
answers
57
views
Why data required for hyperparameter tuning is considered as an additional data?
Any parametric model may have parameters as well as hyperparameters. Learning algorithm deals with parameters and hyperparameters should be dealt outside learning algorithm. Consider the following ...
- 3,099
1
vote
0
answers
39
views
Is it true that real world data is highly discontinuous?
A function $f$ is said to be continuous at a point $c$ if it satisfies three properties:
Should be defined at the point $c$
Left and right-hand limits at $c$ must be equal i.e., the limit must exist
...
- 3,099
0
votes
0
answers
25
views
Adding data to training results in loss random peaks
I have succesfully trained ssd_mobilenet_v2_keras for object detection, with a dataset of about 3700 images. Now I have more images to add. I tried adding only a few images (150-300) to see what ...
- 73
0
votes
0
answers
35
views
Which Neural Network Topology to choose, are Transformers suitable?
I have a regression problem and I am not quite sure which architecture to choose. I never worked with transformers before, but I generally understand how they work and I think they might be suitable. ...
0
votes
1
answer
26
views
Data analysis before feeding to ML pipeline
I'm new to machine learning and I've been working through a dataset of ~3000 records with ~100 features. I've been hand rolling Python and R scripts to analyse the data. For example, plotting the ...
- 103
0
votes
0
answers
16
views
Weight for Samples on SVM (Support Vector Machine)
There is a option sample_weight in fit(X[, y, sample_weight]) function (OneClassSVM, sklearn library).
If I use the option ...
1
vote
1
answer
261
views
Is there a way to parallelise the RL training on multiple stocks to avoid the memory issue?
I have some plans in working with Reinforcement Learning in order to predict the stock price movement. For a stock like TSLA
some training features might be the pivot price values and the set of the ...
- 13
1
vote
0
answers
26
views
Training on the dataset in parts vs training on the whole dataset
What is the difference between these two situations? are they the same ?
#1 : train a model 20 epochs on the whole dataset
#2 : divide dataset into n-parts then train the model 20 epochs on each part
...
- 11
0
votes
1
answer
30
views
An online editor that allows data labeling format [closed]
I have a set of students (~20) that will work on annotating data for an NLP project.
The annotation task will be as in the following:
...
- 123
1
vote
0
answers
60
views
Batch normalization for multiple datasets?
I am working on a task of generating synthetic data to help the training of my model. This means that the training is performed on synthetic + real data, and tested on real data.
I was told that batch ...
- 201
1
vote
2
answers
325
views
Train Validation Test Splitting After or Before Data Augmentation?
I have seen tutorials online saying that you should do data augmentation AFTER doing the train/val/test split. However, when I go online to read some research papers, I see numerous instances of ...
- 21
0
votes
1
answer
31
views
How can I use a prediction model (e.g., ARMA model or LSTM) for multi-variate data?
I have a question
I have had a dataset below
...
0
votes
0
answers
18
views
Comparing results of different image splicing methods on a part of the CASIA 2.0 dataset
So I am working on an image splicing detection algorithm using ResNet-50 model. I am using the CASIA 2.0 dataset which consists of 7491 Authentic images and 5123 Fake images. However out of the fake ...
1
vote
1
answer
36
views
How can I address missing values for LSTM?
I'm a student and writing my first paper for submission on conference. I have a question
there is a dataset below. this is temporal-spatial dataset.
...
0
votes
0
answers
47
views
Simple example for side information
Many research papers contain the phrase "side information". After a search on side information, I got the following definition from here.
In many problems of machine learning and computer ...
- 3,099
1
vote
1
answer
240
views
What is the meaning or implications of the rank of a dataset for machine learning algorithms?
Consider a dataset with $n$ training examples and $d$ features.
Let $D_{n \times d}$ be the data matrix and $r$ be the rank of it.
In matrices, rank $r$ is generally useful in
Knowing the dimension ...
- 3,099